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Executive Summary 

AED conducted a case study of six public high schools in New York City as part of a 
multifaceted evaluation of a small schools initiative funded by the Bill & Melinda Gates 
Foundation. Through surveys, interviews, and focus groups, we gathered information and 
opinions from the schools' principals, teachers, support staff, and students. We also 
conducted observations of classes and meetings to gather data about practices related to 
such vital matters as personalization, classroom instruction, college preparation, and staff 
collaboration. 

The Bill & Melinda Gates Foundation initiative tapped the expertise of intermediary 
organizations that provided direct supports to schools. An important part of this case 
study, then, was to explore how the intermediaries supported schools and how school 
personnel perceived the intermediaries' interventions. 

Guided by the research literature on effective school and dassroom practices, the case 
study looked for evidence of the following practices related to high- guality education: 

Intermediary support. We looked at relationships between the supports provided by 
the intermediary organizations and the effectiveness of the school and dassroom practices 
listed below. 

Personal and academic support. Case- study schools employed a variety of 
strategies— some more comprehensive than others— to attend to the academic and sodal- 
emotional needs of every student and ensure that no student "fell through the cracks." 

Effective instructional practices. We documented the extent to which teachers 
employed spedfic practices that charaderize effedive, academically rigorous, and 
personally and culturally relevant instruction. For example, we documented the 
occurrence of hands-on instruction, metacognitive skill- building, frequent assessment and 
feedback, higher order thinking, student choice, and many other techniques charaderistic 
of effective instruction. Staff input helped us determine fadors that fadlitated and 
hindered implementation of such practices. 

College preparation. Every case- study school aimed to prepare its students for college, 
attending not only to the rigorous instruction that makes students ready for college work 
but also to career and college seledion and to the college application process. 

The nature of case- study research— collecting data from a limited number of schools and 
from a limited number of classrooms, meetings, and people in those schools— means that 
our findings cannot be generalized to other schools funded by the foundation. 
Nevertheless, these findings are consonant with previous research literature and suggest 
fadors to be considered in designing and implementing school- improvement initiatives. 



1. A strong focus on teacher learning and 
collaboration supports effective instructional 
practices. 

The schools in which we saw the most frequent and complete implementation of 
effective instructional practices in classroom observations were also the schools in which 
we found strong evidence of support for teacher learning, resulting in a more integrated 
and coherent approach to instruction. Our findings suggest that quality teacher learning 
is neither a one-time event nor a top-down initiative. Rather, teachers develop their 
professional skills in a culture of inquiry and reflection in which they routinely 
collaborate with one another, participate in the design of their own learning 
opportunities, and develop leadership skills. The schools in which we most frequently 
saw effective, rigorous, and relevant instruction were those supported by an 
intermediary organization that placed teachers within a network of teachers from 
similar schools and empowered them to teach and learn from each other. 

2. An integrated approach to college 
preparation fosters a strong college-going 
culture. 

By design and as part of their mission, all six case- study schools stressed the college- 
going message. However, some schools offered a compartmentalized approach to college 
preparation while others used a more integrated approach. In the compartmentalized 
approach, college preparation took place in silos: teachers wrote recommendations and 
sometimes helped with college essays, while counselors were responsible for 
applications and similar paperwork. The integrated approach involved all staff 
members; teachers joined counselors in motivating students to overcome the barriers 
separating them from college and in helping them with all aspects of selecting and 
applying to schools. 

3. Consistent, structured programs facilitate 
student support. 

Again by design, all of the six case- study schools had organizational features in place to 
monitor and address the academic and social- emotional needs of all students. These 
features were designed to ensure that all students received needed supports and no 
student fell through the cracks. The distinguishing characteristic between what 
appeared to be more and less effective supports was not which features were 
implemented, but to what extent the features were consistent and structured. For 
example, organizational features in which the same staff member worked with the same 
students for all four years of high school offered a more comprehensive web of support. 
Similarly, structured protocols for examining student progress— informed by both 
quantitative and qualitative data— enabled a holistic approach to addressing student 
needs. 
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1. Introduction 



The Bill & Melinda Gates Foundation invested over $ 150 million in a major effort to 
reform New York City high schools. Central to this effort has been the creation of small 
high schools intended to provide high-guality education for low- income, immigrant, and 
minority students who face significant challenges to school success. The foundation 
worked with the New York City Department of Education (DOE) and other partners on 
system- wide efforts to support these new small learning communities. Since 2002, over 
200 such small high schools have been created in some of NYC's most disadvantaged 
neighborhoods. 

While the new schools have a wide variety of themes and educational philosophies, they 
share three common design characteristics (MDRC, 2008): 

• Academic rigor. Schools are expected to be college- preparatory in that they 
move all students toward acquisition of a New York Regents diploma. 

• Personalization. Schools provide academic and social- emotional support to 
each individual student. 

• Partnerships. Schools gain access to resources and expand connections by 
partnering with community and intermediary organizations 

Intermediary organizations, which received funding from the Bill & Melinda Gates 
Foundation for the express purpose of starting new small schools, were key partners 
that provided direct supports to schools. Intermediary organizations are typically non- 



profit organizations that operate between policymakers or funders and the entities, such 
as school districts, charged with implementing new programs and practices (see full 
definition of educational intermediaries on pages 9- 10) . 

This report presents findings from a case study, conducted by AED, of six Gates- funded 
schools in New York City. This study is one of six interrelated studies of the small 
schools initiative. A set of implementation studies includes a descriptive study by Policy 
Studies Associates, Inc. (PSA) of the role intermediary organizations play in designing 
and facilitating implementation of small high schools, as well as a study of school 
characteristics conducted by MDRC. This MDRC study describes the systemic context in 
which these schools were created, the changes in high school options and in student 
enrollment patterns that have occurred, and how the new schools compare with others 
in the system on a wide range of variables. A fourth study, also conducted by MDRC, is 
an impact analysis of small schools on student outcomes using the naturally occurring 
experiment made possible by NYC's use of a lottery system to assign students to 
oversubscribed schools. In addition to the impact and implementation studies just 
described, the foundation commissioned two internal studies: a historical analysis of its 
role in school reform and the lessons learned over time, conducted by Kronley & 
Associates, and a fiscal analysis of the small schools initiative, conducted by the 
Parthenon Group. 



Purpose 

The AED case study complements the impact and implementation analyses by providing 
in-depth images of school and instructional practices in small schools funded by the Bill 
& Melinda Gates Foundation. The study allowed us to look for nuanced patterns related 
to implementation, so that the findings contribute to the field's knowledge about how 

and under what conditions small schools implement best practices. Guided by 
the research literature on school effectiveness and by the foundation's 
programmatic emphasis, the case study focuses primarily on personalization, 
college access, and the instructional rigor that leads to college readiness. 

The literature enabled us to identify key factors and best practices in each of 
these areas. We went into the case- study schools to look for evidence that they 
adopted these practices and to discern, as much as possible, the conditions that enabled 
them to do so. We studied how the intermediaries supported schools to design and 
implement effective practices in these three areas. The findings therefore can help 
inform future school improvement initiatives. 

Specifically, the case study looked for evidence of the following practices related to high- 
quality education: 

• Intermediary support. What guiding principles did the intermediary 

organizations apply in starting these small schools? What kinds of support did 
they provide? Can relationships be discerned between the supports provided 
and the effectiveness of the schools? 



"They treat us like college 
students here." —Student in a 
case -study school 



2 Small High Schools at Work: A Case Study of Six Gates-funded Schools in NYC 



• Personalization and academic support. To what extent and how did 

schools facilitate personal relationships between staff and students? What 
strategies and organizational features did they apply to ensure that the 
academic and social- emotional needs of all students were addressed? 

• Instructional practices. To what extent and how did schools employ 
effective instructional technigues suffused with the academic rigor and personal 
and cultural relevance that lead to student learning? What factors facilitated 
and hindered the design and implementation of effective instructional 
practices? 

• College preparation. How did schools support students in the many 
activities involved in selecting and applying to colleges? How did they prepare 
students for college- level work? 



IVktliods 



Through surveys, interviews, and focus groups, AED gathered information and opinions 
from the schools' principals, teachers, other staff, and students. We also conducted 
observations of classes and meetings to gather data about practices related to such vital 
matters as classroom instruction, staff collaboration, and teacher professional 
development. We tapped MDRC's school characteristics study for additional data on the 
schools' work. PSA's intermediary study helped inform our examination of the ways the 
intermediary organizations supported the schools. The three intermediaries studied 
were among those that received Gates Foundation grants to develop small schools in 
New York Qty: 

• Intermediary A has a strong orientation toward college- readiness and 
effective instruction. 

• Intermediary B has a strong focus on leadership support and using data to 
inform practice. 

• Intermediary C uses a pedagogical approach based on a youth development 
model to serve students who are recent immigrants to the U. S. 

PSA, as part of its data collection, asked each of these intermediaries to identify two 
schools to participate in the case study: one school that was successful in implementing 
the organization's model and another that had struggled but ultimately "turned around." 
The schools and their students are described in Chapter 2. The intermediary 
organizations and their roles in supporting the schools are examined in Chapter 3. 



Findings 



Chapters 4- 6 of this report are devoted to our findings in the three key areas identified 
by the foundation and the literature as factors in small- school success. Chapter 4 
examines the schools' practices in fostering personalization and academic support for 




every student. We found that every school had organizational features in place to 
provide such personal and academic support. What seemed to distinguish the schools 
that were more successful in establishing supportive relations was not so much the 

kinds of organizational features they employed as the 
consistency with which they structured them. 



Fig. 1.1. 12th grade 
physics class at a case 
-study school 



were also schools that— following the lead of their 
intermediary— provided strong support for teacher learning and collaboration. 

Academically rigorous instruction is a major component of college preparation, the 
subject of Chapter 6. Led by teacher and support staff comments, we distinguish 
between college access activities— the many steps required to choose colleges and to 
apply for admission and for financial aid— and college readiness activities, the ones that 
prepare students for college- level work. Effective instruction is thus a primary 
component of college readiness. We found a commitment to college preparation in all 
six case- study schools. Those that employed an integrated approach involving all staff 
members in college access activities seemed to be the ones that were most successful in 
creating a school- wide college- going culture. 



Our findings on effective instructional practices characterized 
by rigor and by personal and cultural relevance are the 
subject of Chapter 5. Effective instruction— the gateway to 
student learning— was, by design, a high priority for every one 
of the case- study schools. Nevertheless, our classroom 
observations, supplemented by interviews with those classes' 
teachers, showed a higher incidence of effective, rigorous 
instruction in some schools than in others. A key finding was 
that the schools in which we found more effective instruction 



Chapter 7 summarizes the most important findings of the case study and suggests 
avenues for further research into what makes small schools effective. These findings 
must be taken as suggestive, not definitive, by the very nature of case- study research. 
We studied only six schools, observing eight classrooms in each and interviewing their 
teachers. We collected a great deal of additional data using surveys, focus groups, and 
the principal interview, but we could reach only a limited number of staff and students, 
who may not have been entirely representative. Equally importantly, we cannot 
establish causal relationships. We can say that we observed, for example, strong 
instructional practices in schools where we also observed strong support for teacher 
learning, but we can only infer that teacher learning was one reason for the presence of 
effective instructional strategies. 



Such inferences are, however, supported by previous research on the factors that lead to 
small- school success. The limitations of the case- study method are balanced by its 
strengths. We spent days in each school, gathering data using multiple methods, both 
quantitative and qualitative, that can be "triangulated" to yield a reasonably complete 
picture of each school's strengths and challenges. We spoke with numerous 
individuals— principals, teachers, support staff, and students— using open-ended 
protocols that allowed them to bring up points we might not have considered in 
constructing questions. The result is a finely grained portrait of each school that yields 
tantalizing suggestions of what works and what doesn't in designing and implementing 
small, personal, academically rigorous schools. 
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2. Methods 



AED's case study of six high schools funded by the Bill & Melinda Gates Foundation was 
grounded in knowledge about practices that characterize instruction in schools that are 
high-performing or improving, especially those that serve young people of color and 
students who are economically disadvantaged. Specifically, we used the research to 
identify best practices at the high school level and then designed our data collection to 
obtain data on the extent to which we saw evidence of these practices in the case- study 
schools. Through multiple data collection strategies, we developed detailed portraits of 
practice and analyzed the data for patterns that suggested factors that facilitated or 
hindered implementation of practice. The data are correlational, not causal. Therefore, 
our findings are suggestive, rather than confirmatory; they are not intended to be 
generalized to all Gates- funded small schools. These findings may lead to hypotheses 
that can be confirmed through additional study. 



Data Collection 



AED collected data in Fall 2008 and Spring 2009 using the following methods, each of 
which is described below: 



• I n- depth principal interviews 

• Classroom observations 

• Follow- up teacher interviews 

• Teacher and support staff focus groups 

• Student focus groups 

• Teacher survey 

• Meeting observations 

• Data from other sources: related studies of Gates- funded schools, document and 
artifact review 

Instruments used to collect data are reproduced in Appendix A. 



Principal Interviews 

At the beginning of our data collection, we conducted an hour-long 
interview with each principal covering topics related to the school mission 
and vision, how the school provided academic and social- emotional 
support to students, how the school prepared students for college, and 
what kinds of support were provided by the school's intermediary 
organization. Though we used a structured interview protocol, these 
interviews share the limitations of any self-reported data, including 
participant bias or self- censorship. In addition, though guestions were 
designed to gather information about the school's entire history, principals 
are likely to have mentioned the most recent or most salient events when 
describing, for example, the intermediaries' support. 



"The mission is for students 
to learn to think critically. . . 
and prepare for college and 
other educational 
opportunities after high 
school. " —Principal interview, 
School 5 



Classroom Observations 

We conducted observations in eight classrooms in each school, reaching approximately 
one- third of the instructional staff. The teachers were selected for observation by the 
researchers working alone or with the administrative staff to include a diverse sample in 
terms of grade level and subject area. Half of the observations were conducted in the 
lower grades ( 9 th or 10 th ) and half in the upper grades ( 11 th or 12 th ) . The observations 
represented the four core subject areas, with two at each school conducted in English, 
mathematics, science, and social studies classes. Wanting to get a sample of classes that 
reflected the average student, we did not observe self-contained special education 
classes. 

We used a semi- structured observation protocol with a mix of open-ended and dosed- 
ended questions. Researchers were trained extensively on the use of the protocol. To 
obtain data on inter- rater reliability on the closed- ended items, two researchers 
observed one or two dassrooms together per school. Each answered the questions 
separately, and we compared responses. We obtained strong inter- rater reliability scores 
on most items, ranging from .5 to .9 (kappa value) on each item. 
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The classroom observation data are limited by the fact that they represent a single point 
in time, on the day of the observation. We cannot determine the extent to which the 
lesson we observed is representative of all lessons given by that teacher. Teachers and 
students may also have been influenced simply by our presence in the room. In addition, 
because teachers were selected for observation through a convenience sampling 
procedure, we do not claim that they are representative of all of the teachers in the 
school. 



For each classroom observed, we conducted a follow-up interview with its teacher. 

These in-depth interviews allowed us to ask about the planning and context of the lesson 
we had just observed as well as about instructional practices that we may not have been 
able to observe, such as differentiated instruction. We also asked about school policies 
and practices, particularly those pertaining to teacher professional development and 
collaboration, college readiness, and support from the intermediary. The interviews 
provided rich data but are subject to the same limitations as the principal interviews. 



We conducted two (and in one case three) focus group interviews in each school: one 
comprising teachers and the other comprising support staff such as guidance 
counselors, social workers, community liaisons, and parent coordinators. All teachers 
and staff were invited to participate in the focus groups; the first eight who volunteered 
were selected. (In some cases, fewer than eight volunteered.) Topics discussed included 
how the school prepared students for college and how the intermediary supported the 
instructional and support staff. 

Facilitators of these and the student focus groups used a specific protocol that guided 
discussion while allowing participants to propose and pursue topics of their own. 
However, self- selected participants may not have been representative of the total 
population in the school. 



We conducted three student focus groups in each school, all at different 

grade levels, with at least one at the lower grade level ( 9 th or 10 th ) and at 

least one at the upper grade level ( 11 th or 12 th ) . Students in selected 
classrooms or advisories representing the grade levels of interest were invited to 
participate; only those who returned parent consent forms were selected. If more than 
eight students at a particular grade level returned parent consent forms, we selected 
students on a first- come, first- served basis. The focus groups were conducted during 



Follow-up Teacher Interviews 



Teacher and Support Staff Focus Groups 



Student Focus Groups 



"We have great relationships 
with our teachers, and they have 
really pushed us. They have high 
expectations for us. " —Student 
focus group, School 4 
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students' lunch or advisory periods or after school. Topics discussed included academic 
rigor and challenge in classroom instruction as well as college preparation. 



Teacher Survey 

AED, PSA, and MDRC co- developed a teacher survey to be piloted in the case- study 
schools, with the intention that the survey may be administered citywide at a future 
date. The survey was administered to case- study teachers by AED in February and 
March 2009. We collected 143 surveys, representing an 86% response rate. Topics on 
the teacher survey included school climate, teacher professional development, college 
preparation activities, and, to some extent, instructional practices. Many of the items 
included on this survey were adapted from the Chicago High School Redesign Initiative 
teacher surveys by the Consortium on Chicago School Research (2005, 2007). The 
teacher survey data are limited by the inherent possibility for bias in self-reported data; 
teachers may have given socially desirable responses or inaccurately recalled the 
freguency and nature of their practices. 



Meeting Observations 

AED observed at least three teacher- team or professional development meetings per 
school. Using a semi- structured protocol, we looked for examples of teacher 
collaboration, use of student data, and discussion of student supports and college 
preparation activities. This protocol and the other instruments used in the case study 
are collected in Appendix A. The meeting observations are subject to limitations similar 
to those of the classroom observations: the possibility that the meetings were not 
representative and the "observer effect." 



Data from Other Sources 

In addition to the data AED collected, we reviewed and analyzed data from the principal 
survey administered by MDRC in spring 2008, the intermediary interviews conducted 
by PSA, the internal- to- the- foundation fiscal analysis conducted by the Parthenon 
Group. We also examined documents and artifacts collected from the schools, such as 
parent and student handbooks, printed mission statements, meeting protocols, lesson 
plans, and principal newsletters. The principal survey, collected from each principal in 
our study, 1 provided data on school policies and practices that supported student 
achievement and college preparation as well as on the principal's perception of the 
emphasis and helpfulness of the support provided by the intermediary. The survey is 
reproduced in Appendix A. The intermediary interviews covered topics including the 

1 A low response rate to the principal survey meant that it could not be used for other studies, as 
intended. However, the principals of all six case- study schools responded to the MDRC survey. 
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intermediary's model and the services provided to schools. The fiscal analysis provided 
an overview of the intermediary's start-up, operations, and program components. 



Data Analysis 

Our approach to analyzing this data was a grounded- theory approach. Based on our 
observations and interview data, we identified patterns and developed theories on 
differences in implementation of effective practices. Once the patterns and theoretical 
concepts were identified, we revisited the data to verify (or discount) the patterns and 
theories and to describe the phenomena. 

In reporting findings from the data, we identified general patterns and provided specific 
examples that illustrated those patterns. To the extent possible, we quantified responses 
from individuals and instances of evidence of particular practices in the classroom 
observations. Because the teacher, support staff, and student focus groups were 
conducted as small- group discussions, rather than systematically asking each 
respondent the same question, data from the focus groups were summarized, with a 
note as needed about the extent to which the entire group agreed or disagreed with 
statements made by individuals. Similarly, data from meeting observations, because 
data were collected through observation rather than systematic questioning, were 
summarized. 

In some cases, themes emerged from the data that were not specifically explored on our 
protocols and were therefore not collected from each school. The project timeline did 
not allow us to return to the schools to ask systematically about these patterns. Although 
we were therefore not able to determine the extent to which such practices or themes 
were present or true for every school in our study, we report them here if they are 
relevant to our research questions. 



Sample 

The case study was based on a sample of three intermediary organizations and two 
schools from each intermediary. 



Intermediaries 

Education intermediaries, according to the definition developed jointly by researchers 
involved in the six interrelated studies of the small- schools initiative, are typically 
nonprofit organizations that operate between policymakers or funders and the entities, 
such as schools and school districts, charged with implementing new programs and 
practices. In the efforts of the Bill & Melinda Gates Foundation to create small high 



schools, intermediaries often served both as fiscal agents for distributing grant funds to 
schools and as sources of experience with and advice about creating, incubating, and 
operating small schools. This advisory support included leadership development, 
instructional support, and college readiness services. To varying degrees, 
intermediaries also advocated for their schools with local educational authorities, helped 
identify and procure facilities, connected schools with other organizations and grantee 
networks, nurtured such networks, assisted with operations and staff selection, provided 
teacher professional development, and facilitated program planning, development and 
implementation. 

The intermediaries in this case study were selected from among 18 intermediaries that 
received grants from the Bill & Melinda Gates Foundation to develop small schools in 
New York City. These intermediaries, their emphases, and the supports they provided 
to schools are explored in Chapter 3. 



Schools 

In interviews conducted by PSA, researchers asked leaders of each of the three 
intermediaries to nominate two schools to be the subjects of our study: one school that 
was considered successful in implementing the organization's theory of change and 
another that had struggled but ultimately "turned around" and was considered to have 
been successful in implementing the model. The intermediaries used their own criteria 
to define "strong implementers" and "turn-around schools." In order to be consistent 
with the data samples from MDRCs impact and school characteristics studies, we 
considered only schools serving grades 9- 12 that determined enrollment through the 
district choice system. 

Intermediaries A, B, and C nominated Schools 1, 3, and 5, respectively, as "strong 
implementers." The "turn-around schools" in our sample were School 2 for 
Intermediary A, School 4 for Intermediary B, and School 6 for Intermediary C. 

The "turn-around" schools faced, in their early years of development, the kinds of 
challenges that are often endemic in large urban school districts. School 2 faced 
difficulty in finding a permanent location and had moved four times in four years. Its 
staff was very inexperienced and went through a great deal of turnover. The turnover 
extended to the principal position, where the incumbent changed every year; the 
principal we interviewed had taken over in 2007- 08. School 4 also initially struggled 
with principal turnover, with the principal at the time of our study being the third in the 
school's four- year history. School 6, when it first opened, was placed by the NYC DOE in 
a dangerous neighborhood with few newly arrived immigrants, the school's target 
population. As a result, the school was under- enrolled. At the time of our study, the 
school had relocated to a neighborhood with a new- immigrant population. However, 
parents also considered the new neighborhood dangerous, according to the principal. 

Table 2.1 (next page) shows teacher and instruction- related characteristics of the case- 
study high schools. For comparative purposes, the final column shows the eguivalent 
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data for small high schools in NYC that, like the case- study schools, are not academically 
selective. 



Table 2.1. Teacher and Instructional Characteristics of Case-Study Schools, 2007-08 




Intermediaiy A 


Intermediaiy B 


Intermediaiy C 


All Small 

Academically Non- 
Selective Schools 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Year school opened 


2004 


2005 


2003 


2003 


2004 


2005 


n / a 


Current principal's tenure (years) 


5 


2 


6 


4 


5 


4 


n/a 


Teaching out of certification 


9% 


35% 


12% 


12% 


13% 


24% 


19% 


Teachers with fewer than three 
years of experience 


6% 


47% 


27% 


8% 


34% 


53% 


36% 


Teachers with master's degree plus 
30 hours or doctoral degree 


22% 


29% 


39% 


48% 


19% 


12% 


24% 


Overall School Quality Review (SQR) 
2 

score (out of 5) 


N/A 


N/A 


3 


4 


4 


4 


3.4 


SQR Statement 1 : Gather Data 


N/A 


N/A 


4 


4 


4 


4 


3.5 


SQR Statement 2: Plan and Set Goals 


N/A 


N/A 


4 


4 


4 


4 


3.5 


SQR Statement 3: Align Instructional 
Strategy to Goals 


N/A 


N/A 


3 


4 


4 


4 


3.5 


SQR Statement 4: Align Capacity 
Building to Goals 


N/A 


N/A 


3 


4 


4 


4 


3.7 


SQR Statement 5: Monitor and 
Revise Goals 


N/A 


N/A 


3 


3 


4 


3 


3.3 


NYCDOE Progress Report Letter 
Grade 3 


A 


N/A 


B 


A 


A 


N/A 


B 


SOURCES: Year school opened is from the foundation. Principal tenure data are from the principal survey. Teacher characteristics and 
school accountability data were provided to AED by MDRC. Calculations were made from the New York State Report Card and data on 
new small schools provided by the DOE for school years 2007-2008, and DOE accountability measures including the New York City 
Progress Report, School Quality Review. School 2 was categorized by MDRC as a medium-sized school because the 9th grade enrollment 
was just over its maximum of 175. Similarly, Schools 5 and 6 were categorized by MDRC and the New York City Department of 
Education as academically selective because they enroll only students who are new immigrants and eligible for ELL services. However, 
for the purpose of describing the case-study schools, we believe it is appropriate to compare these schools to small academically non- 
selective schools. 



2 The School Quality Review (SQR) overall score and five statement scores were converted from a 
nominal ranking scale to a numeric scale so that calculated averages could be shown in this table. 
MDRC converted the SQR's nominal scale to a five- point scale: "underdeveloped" = 1, 
"underdeveloped with proficient features" = 2, "proficient" = 3, "well developed" =4, and 

3 The DOE computed progress report scores for schools serving all grades 9- 12. These schools 
received scores along three dimensions: school environment, student performance, and student 
progress. 
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The case- study schools opened between 2003- 04 and 2005- 06, with two schools 
seeing their first graduating class during the time we conducted our research. In four of 
the six schools, the principal at the time of our study had been in place during the entire 
life of the school. The other two schools had experienced principal turnover, as noted 
above. 

Table 2.1 shows a fair amount of variety in the credentials and experience of teachers in 
the case- study schools. In Schools 2 and 6, a substantial proportion of teachers— more 
than one- third in School 2— were teaching outside their area of certification. In the other 
four schools, the percentage of staff teaching out of certification was lower than the 
average for small non- selective schools. Further, in Schools 2 and 6, about half of the 
teaching force had fewer than three years of experience. In Schools 3 and 5, about one- 
third of teachers had fewer than three years of teaching under their belt— somewhat less 
than the average for small non- selective schools. Schools 1 and 4, by contrast, had far 
fewer inexperienced teachers. The data also show that in four of the six schools, fewer 
than one- third of the teaching staff had advanced credentials, defined as a master's 
degree plus 30 hours or a doctorate— the exception being the two schools served by 
Intermediary B, where 39% and 48% of teachers had advanced credentials. 

Data from the DOE accountability system show that the case- study schools scored high 
on indicators of quality as measured by the School Quality Review ( SQR), with each of 
the four schools with data scoring in the "well developed" or "outstanding" categories 
overall and in each sub- category of the review. Further evidence that these are high- 
performing schools— relative to their peers— comes from the DOE's progress report. The 
report card gives a letter grade to schools based on their performance in comparison to 
schools serving similar students. Three of the schools with data scored an A, and one 
scored a B. These data indicate that, in comparison to their peer schools serving a 
similar population of students, the case- study schools performed well by DOE 
accountability standards. 



Students 

Students in the case- study schools were generally characteristic of students in NYC 
public high schools and specifically of those in all small, non-selective schools. Table 2.2 
(next page) shows the demographic and academic characteristics of first- time 9 th 
graders in the case- study schools and in comparable schools, using data compiled by 
MDRC (2009). 

Table 2.2 shows that the case- study schools served predominately African-American and 
Latino youth. In School 5, one of those serving immigrant youth, about one- fourth of 
students were Asian. A large proportion of entering 9 th graders in the case- study schools 
were overage for eighth grade, ranging from 15% for School 1 to 52% for School 6. Most 
of the students in these schools came from low- income families, as indicated by the high 
percentage of students eligible for free or reduced- priced lunch: more than 80% in all 
schools but one. Schools 5 and 6 had the largest percentages of both overage and low- 
income students. 
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Table 2.2. Demographic and Performance -Based Characteristics of First-Time 9th 
Grade Students Enrolled in Case-Study High Schools, 2007-08 




Intermediary A 


Intermediary B 


Intermediary C 


All Small 

Academically Non- 
selective Schools 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


9 th grade enrollment 


114 


100 


129 


108 


125 


66 


n/a 


Total school enrollment 


373 


310 


467 


434 


417 


227 


n/a 


Gender 


Female 


50% 


49% 


48% 


55% 


45% 


45% 


49% 


Male 


50% 


51% 


52% 


45% 


55% 


55% 


51% 


Ethnicity 


White 


3% 


0% 


3% 


0% 


7% 


12% 


3% 


Black 


63% 


36% 


85% 


44% 


26% 


49% 


45% 


Hispanic 


33% 


61% 


11% 


55% 


40% 


29% 


49% 


Asian 


1% 


2% 


2% 


1% 


27% 


9% 


3% 


American Indian 


0% 


0% 


0% 


0% 


1% 


0% 


0% 


Overage for eighth grade 


15% 


29% 


17% 


29% 


41% 


52% 


28% 


Eligible for free and reduced-price 
lunch 


80% 


83% 


46% 


80% 


92% 


92% 


82% 


Special education 


12% 


14% 


14% 


19% 


0% 


0% 


14% 


English language learner 


3% 


14% 


5% 


14% 


76% 


56% 


14% 


Average 8 th grade math proficiency 4 


Low 


46% 


62% 


50% 


57% 


53% 


77% 


53% 


Medium 


38% 


32% 


39% 


38% 


29% 


23% 


37% 


High 


17% 


7% 


10% 


5% 


18% 


0% 


10% 


Average 8 th grade English language arts proficiency 


Low 


31% 


58% 


42% 


62% 


94% 


94% 


52% 


Medium 


58% 


36% 


50% 


32% 


6% 


6% 


40% 


High 


11% 


7% 


8% 


5% 


0% 


0% 


8% 


SOURCES: MDRC calculated data from DOE October enrollment data for first-time 9th graders in school year 2007-08, and High 
School Application Processing System (HSAPS) data for 8th grade students in 2006-07. 

NOTE: Rounding may cause discrepancies in sums and differences. 



In four of the case- study schools, the percentage of students eligible for special 
education services ( 12% to 19%) was similar to the percentage in all small academically 
non- selective schools (14%). According to DOE data, the two schools whose target 
population was new immigrants. Schools 5 and 6, did not enroll special education 
students. However, since these data are reported for the 8 th grade, some students in 

4 New York State defines four proficiency categories— levels 1 through 4. For this study, these have 
been recoded into three categories: low, medium, and high. Low proficiency is defined as scoring 
in proficiency level 1 or the bottom half of proficiency level 2. Medium proficiency is defined as 
scoring in the higher half of proficiency level 2 or the lower half of proficiency level 3. High 
proficiency is defined as scoring in the higher half of proficiency level 3 or proficiency level 4. 
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these schools may have classified into special education at a later date. According to the 
principal at school 6, students are in the NYC DOE system at least two years before 
being classified as special education. Because students in these schools have been in the 
country for less than four years, very few would have been classified as special education 
by the 8 th grade. 

Those same two schools, not surprisingly, served by far the highest proportion of 
students eligible for English language learner (ELL) services, at 76% and 56%. Schools 1 
and 3 had very few ELL students, while 14% of students at both Schools 2 and 4 were 
ELL students. Finally, the data show that large proportions of entering 9 th graders in 
these schools had low levels of proficiency in math and English language arts, indicating 
that they entered high school with serious academic deficiencies. The percentage of 
students with low ELA proficiency is extremely high in the two schools serving new 
immigrants. 

Taken together, the high percentage of students who were overage for grade, eligible for 
free lunch and ELL services, and at low levels of math and English proficiency upon 
entrance to high school point to a population that is distinctly more disadvantaged in 
the two schools served by Intermediary C than in the other case- study schools. 
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3. The Intermediary 
Organizations 



Informed by the literature on intermediary organizations (Honig & Hatch, 2004; 
McDonald & Klein, 2003; Rogers, 2007), teacher professional development (Ball & 
Cohen, 1999; Cohen & Hill, 2001; Darling- Hammond, 1993; Little, 1990; little, 1999), 
and school- community relationships (Hirota, 2004; Honig, Kahne, &McLaughlin, 
2001; National Research Council & Institute of Medicine, 2004), our case-study 
research considered the ways in which intermediaries supported the growth and 
ongoing development of case- study schools. Throughout our case- study interviews, 
surveys, and focus groups, we asked principals, teachers, and guidance counselors to 
describe the nature and frequency of their work with their intermediary organizations. 
As we analyzed and sought resonance among various data sources, we developed a 
descriptive picture of the relationships between intermediaries and schools. 

This chapter begins to paint this picture by exploring the emphases of the 
intermediaries' work, using both the organizations' self- descriptions and data from the 
principal survey and case- study research. We then describe the specific supports 
intermediaries provided to schools. 



Emphasis of the Intermediaries' 
Work 



The three intermediary organizations chosen for this study differ in significant ways. 
They were created under different circumstances and operate with different business 
models and theories of change. For a detailed description of all of the intermediaries 
funded by the Bill & Melinda Gates Foundation to start new small schools, see the Policy 
Studies Associates report (2009). A brief description of each intermediary serving case- 
study schools follows. A richer portrait emerges when we next examine the case- study 
schools' perceptions of the emphases of the intermediaries' work 



Intermediaries' Descriptions 

Intermediary A 

Intermediary A partners with schools and districts to create autonomous small schools 
and to transform large comprehensive high schools into academically rigorous, 
personalized learning environments that prepare students for success in college and 
beyond (PSA, 2009). The intermediary collaborates with schools to create 
comprehensive plans and implementation strategies customized to meet the unique 
needs of schools and districts. The organization's work is guided by a set of research- 
based principles of effective practice. A partner organization supports the intermediary 
and schools with supports and services such as mathematics and writing assessments, 
formative data for planning purposes, documentation of schools' implementation 
efforts, and recruitment and training of instructional coaches. Coaches provide 
individualized support to schools, spending at least one day per week in the school. The 
intermediary also provides professional development institutes. 



Intermediary B 

Intermediary B is a school reform organization that is guided by research- based 
principles of effective school design. This intermediary emphasizes the use of data to 
inform practice, providing schools with tools and data for monitoring student 
achievement and communicating attendance and graduation rate goals to all schools in 
its network Intermediary B also provides coaches to the schools, with a particular focus 
on leadership development. It offers professional development institutes and an 
extensive set of online tools, materials, and resources that are open to the public. Unlike 
the other case- study intermediaries, this intermediary connects schools with 
community- based partners who serve as the fiscal agents for Bill & Melinda Gates 
Foundation funding and support schools in their reform efforts. 
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Intermediary C 

Intermediary C develops a network of small public high schools that feature an 
interdisciplinary, student- centered environment. In this intermediary's model, students, 
all of whom are recent immigrants from other countries, work collaboratively in small, 
heterogeneous groups. The intermediary emphasizes leadership development and uses a 
network model in its efforts to help schools improve. Teachers, support staff, and 
administrators are encouraged and supported in efforts to collaborate and share 
practices across schools. The organization's website offers tools, materials, and 
resources for staff including curricula and videos of practice. Intermediary C also 
provides leadership coaching, retreats for principals and assistant principals, and 
professional development institutes. 



Schools' Perceptions of Intermediaries' Work 

A richer, more nuanced picture of the emphases of the intermediaries' work emerges 
when we explore the ways in which principals and teachers described the focus, extent, 
and effect of the intermediaries' support. 

MDRC's principal survey asked about the extent to which intermediaries 
focused on various areas in their work with schools. As shown in Table 3. 1 
(next page), the intermediaries had somewhat different emphases, and 
sometimes principals with the same intermediary reported different levels of 
emphasis in the same areas. For example, the principal of one Intermediary 
A school reported that planning the school budget was not a focus of the 
intermediary's work with the school, while the other said it was an extensive 
focus. This finding may indicate that the intermediaries customized their 
support to schools based on need. In some areas where schools did not 
report an emphasis of the intermediary's work, the area may nevertheless 
have been a focus for the school. For example, though some principals said 
that developing a personalized environment was not a focus of their 
intermediaries' work, we observed that all case- study schools placed a great 
deal of emphasis on personalization. 

Principals of schools served by all three intermediary organizations said that 
various aspects of professional development for teachers and administrators 
were an extensive or moderate focus of their intermediary's work A 
difference in emphasis emerges in the use of instructional coaches. 

Principals of Intermediary A and B schools saw instructional coaches as a moderate or 
extensive focus, while Intermediary C principals perceived instructional coaches to be of 
small or no focus. As discussed in more detail below, our interview and focus group data 
confirmed these findings, revealing, in particular, that instructional coaches provided 
significant supports to teachers in Intermediary A schools. All six principals also 
reported that teacher collaboration was an extensive or moderate focus of their 
intermediaries' work, though our case- study data reveal, as discussed in Chapter 5 on 



" One of our challenges is 
that our school is 100% ELL 
with a specific model— so even 
the state doesn't understand 
how we work. That's where 
[Intermediary C] helps us. 
We're all in the same boat. 
Schools are team-based and 
collaborative, and 
[Intermediary C] is too. They 
bring us together to share 
what we've learned. We share 
problems that we have had 
and how we overcome them. " 
—Counselor, School 5 (focus 
group) 
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Table 3.1. Focus of Intermediaries' Work with Schools 

Where both principals gave the same answer, that answer is shown only once. 




Intermediary A 


Intermediary B 


Intermediary C 


Providing professional development for teachers 


Extensive 


Extensive 

IVbderate 


Extensive 


Planning professional development for teachers 


Extensive 

* 


IVbderate 

Extensive 


Extensive 

IVbderate 


Providing professional development for administrators 


Extensive 

Moderate 


Extensive 


Extensive 

IVbderate 


Providing instructional coaches for teachers 


IVbderate 

Extensive 


Extensive 


Small focus 
Not a focus 


Fostering teacher collaboration 


IVbderate 

Extensive 


IVbderate 

Extensive 


Extensive 


Developing a personalized environment 


IVbderate 

Extensive 


Not a focus 
Extensive 


Not a focus 


Planning the school budget 


Not a focus 
Extensive 


IVbderate 

Extensive 


Not a focus 


Developing organizational structures 


IVbderate 

Extensive 


Not a focus 
IVbderate 


Small focus 


Using data to inform practice 


IVbderate 

Extensive 


Extensive 


Not a focus 
Small focus 


Engaging all students in an academically rigorous 
curriculum 


IVbderate 

Extensive 


IVbderate 

Extensive 


Extensive 
Small focus 


Planning college preparation activities 


IVbderate 

Extensive 


Extensive 


Extensive 

IVbderate 


Help map how individual classes will function in a 
progression to build the knowledge & skills students 
need for postsecondaiy education 


IVbderate 
Small focus 


* 

Extensive 


Not a focus 


Planning support services for students 


IVbderate 

Extensive 


Moderate 

Extensive 


Small focus 


*Mssing data 
Source: Principal survey 



instruction, that intermediaries varied in the extent to which they provided structured 
support for cross- school teacher collaboration. 

Principals varied in their assessment of the strength of the intermediaries' focus on 
personalization. Principals of Intermediary A schools saw personalization as a relatively 
strong focus, one Intermediary B principal saw it as an extensive focus while the other 
said it was not a focus, and the two Intermediary C principals said that personalization 
was not a focus. Despite not being a focus of the intermediary, our case- study data show 
that the instructional teams in the two Intermediary C schools seemed to provide a 
strong web of support for individual students, as discussed in Chapter 4. 

Principals of the Intermediary A and Intermediary B schools describe use of data to 
inform practice as an extensive or moderate focus of the intermediary's work. Our 
interview and focus group data confirmed Intermediary B's extensive focus on using 
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student data to inform practice and Intermediary A schools' use of qualitative data, such 
as portfolios of student work, to inform their practice and planning. Though 
Intermediary C principals described use of data as not a focus or a small focus, we saw 
examples of teachers using qualitative data to inform practice and planning. 

All six principals reported that their intermediary's work emphasized college 
preparatory activities to a moderate or extensive degree, a finding confirmed by our 
interviews and focus groups. All but one described academic rigor as an extensive or 
moderate focus. These findings are significant but not surprising given the emphasis of 
the foundation and of the intermediaries on college access and readiness. As we describe 
in Chapter 5 on student learning, intermediaries varied in the ways they supported 
schools in carrying these emphases into the classroom. 



In comparing the intermediaries' stated missions and principles, the principals' report 
of the focus of the intermediaries' work in their schools, and data we collected through 
interviews and focus groups with school staff, we see common threads, which are 
outlined below and explored throughout this report. 

Intermediary A emphasized teacher professional development and improving teacher 
learning, as well as strategies for providing a web of support for students. Teachers and 
principals spoke highly of Intermediary A's professional development opportunities and 
particularly of the value of the instructional coaching provided by the intermediary, 
emphasizing the impact of those activities on their practices. Staff interview and focus 
group data suggest that teacher collaboration and leadership development were 
emphasized in Intermediary A schools. This intermediary's emphasis on building a 
safety net of support for students was evident in the structures, such as advisory groups 
and attendance meetings, that the schools implemented to facilitate the involvement of 
all staff in ensuring that students' academic and social- emotional needs were met. 

Priorities in Intermediary B's work included partnerships with local community 
organizations and use of data. In one of the Intermediary B case- study schools, we saw 
evidence of strong community partnerships. Staff generally described these partnerships 
as beneficial, though some indicated that not all teachers benefitted to the same degree. 
Intermediary B's emphasis on use of data to inform practice was evident in its regular 
provision of student data— including such measures as attendance, grades, and Regents 
passing rates— to schools. In both case- study schools served by this intermediary, we 
saw a clear emphasis on using this data to inform practice, plan interventions, and 
monitor progress. 

Intermediary C appeared to place a great deal of emphasis on teacher professional 
development and collaboration. Indeed, this emphasis was reflected in the extent to 
which we saw strong structures for teacher learning and collaboration in Intermediary C 
schools (see Chapter 5). Our research suggests that this intermediary was primarily 
focused on improving practice through teacher learning and inquiry, using a variety of 
structures such as professional development retreats, inter- visitations, and study groups 
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across schools. In focus groups and interviews, several teachers described the ways in 
which the variety of avenues for formal and informal sharing across schools increased 
the institutional capacity of both schools and teachers. This intermediary also involved 
teachers in designing and leading professional learning opportunities. 



Supports Provided to Schools 

Our data reveal descriptive images that illustrate how intermediaries provided supports 
to case- study schools, specifically, how intermediaries worked with schools to: 

• Provide start- up assistance in conceptualizing and developing new small schools 

• Provide ongoing leadership development for principals 

• Negotiate district policies and advocate for schools with outside organizations 

• Access resources through relationships with community organizations 

• Support the ongoing learning and capacity building of teachers 

• Provide tools to help improve school and classroom practices 

Table 3.2 shows specific kinds of supports each intermediary provided to schools. The 
list is not exhaustive. The data in this table and supporting text reflect only what 
administrators and staff described in surveys, interviews, and focus groups. Therefore, 
the types of support mentioned are likely to reflect the most salient or most recent 
intermediary interactions. 



Table 3.2. Intermediary Supports to Schools 




Intermediary A 


Intermediary B 


Intermediary C 


Start-up supports 


I 


/ 


f 


Assistance with hiring new staff 


f 


/ 


/ 


Mentors/ coaches for principals 


/ 


f 


/ 


School planning teams 


/ 


f 


/ 


Regularly scheduled principals' meetings 


/ 


f 


/ 


Policy negotiation 


/ 


f 


/ 


Establishing relationships with community 
organizations 




I 


I 


PD: Instructional coaches for teachers 


/ 


I 




PD: Conferences 


/ 


f 


I 


PD: Inter-visitations 




f 


f 


PD: Teacher study groups 






/ 


Tools 


/ 


I 


f 


Source: teacher and principal interviews, teacher and support staff focus groups 
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Start-up Support 

Our study reveals that intermediaries offered various types of support to leaders during 
their schools' start-up phase. Across all case- study schools, principals described how 
intermediaries helped with hiring and training new staff, provided coaching or 
mentoring, established school planning teams, and developed relationships with parents 
and community members. 

The principal of School 1 described in an interview the importance of the assistance of 
the Intermediary A instructional coach during the start-up phase of the school: "It was 
great to have [the coach] available when we were selecting the staff and starting the 
school." In the other Intermediary A school. School 2, the principal at the time of our 
study was the school's fourth in four years, so she was not involved with the school when 
it opened in 2005. 

In School 4, the principal described his tumultuous first year after the school's founding 
principal was fired. His interview revealed critical supports he received from 
Intermediary B: 

That first summer, a lot needed to be accomplished, and [Intermediary B] was 
incredibly supportive in helping to secure support from the region, improve 
parental support, and begin to get the academics in order. [Intermediary B] 
rehired and switched around staff so they would be able to focus more heavily 
on academics. In addition, that summer and fall, with [Intermediary B] support, 
we used the Gates money to provide one-on-one support for kids who were 
under- credited, to help them catch up. 

The principal went on to say that, during his first year, the Intermediary B liaison was 
'living" at the school. In their interviews, teachers at this school described their 
perceptions that the intermediary provided substantial support to the school's 
administration. 

The principal at School 3 described how Intermediary B helped to create a school 
planning team and provided critical supports focused on all aspects of running the 
school: "They provided guidance on professional development, assisted the school 
planning team early on, and helped address guestions concerning community and 
parental involvement." 

Principals at Intermediary C schools emphasized the critical role of planning teams and 
the support they received from mentors and other principals in the network of schools. 
In an interview, the principal of School 6 described his 
participation in a study group focused on deepening his 
knowledge of Intermediary C core principles (see box). 

Both principals of schools served by this intermediary 
spoke about the critical role of their school planning 
teams, consisting of a principal mentor, two teachers, and 
one representative from Intermediary C. They described 
how the planning team supported the hiring and training 
of new teachers. The principal of School 6 described the 
support he received from other principals in the network: 



"The year before my school opened, I met with 
another new principal and with experienced 
[Intermediary C] mentors to form a study group 
where we read a series of texts about 
scaffolding and language development. We 
learned how to make the core principles come 
to life and how to achieve faculty buy-in. " — 
Principal, School 6 (interview) 
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[Other Intermediary C principals] actually invited their faculty members to 
leave to join the faculty of my school. I was surprised that principals were so 
willing and didn't feel like we were poaching their staff. Two of my five new 
teachers came from sister schools. They came in already knowing how to teach 
in an [Intermediary C school] . It wasn't just me at those first professional 
development events saying 'This is how you teach at an [Intermediary C 
school] . " I also had those two teachers who led the effort of training new staff. 

This principal attributed his peers' generosity to the intermediary's work to foster 

professional community. 



Ongoing Leadership Development 

Principals described the support that they received from their intermediaries as 
extending beyond the start-up phase and focusing on leadership development. Several 
principals described ongoing meetings with mentors who provided critical support as 
the principals faced new challenges in leading their developing schools. The principal at 
School 2 described in her interview how her intermediary's instructional coach helped 
her to incorporate the school's vision into the curriculum. 'The leadership coach helped 
me focus on instruction, aligning the principles [of Intermediary A's model] to 
instruction and helping me promote inguiry." The principal noted that this support was 
particularly helpful because most of her staff were new and inexperienced teachers. 

Intermediary B also provided mentors who worked with principals at both case- study 
schools. In addition. Intermediary B offered a leadership program, which provided 
principals with mentors as well as seminars and workshops, apprenticeship experiences, 
and inter- visitations. In some cases, intermediaries sponsored events to bring principals 
and mentors together; the principal of School 3 described how Intermediary B 
facilitated “roundtable dinners where principals and the mentors would meet." 

Intermediary C held monthly principals' meetings to provide opportunities for peer 
support and critical reflection. The principal of School 6 spoke of the value of these 
meetings: 

It is the only place where I can talk openly with my peers about the challenges I 
face. They know them, because they work in schools with the same design. They 
have seen all of the same issues before. Next week, I 'm doing a school walk- 
through here with the other principals, and I 'm bringing them into the 
classrooms where teachers are struggling. I feel safe with them, and I know that 
they can brainstorm to help me come up with strategies to improve instruction 
in my school. 

In addition to monthly meetings, we found that intermediaries also offered regularly 
scheduled leadership retreats for principals. Intermediary A supported a leadership 
network by holding meetings for principals and assistant principals during its summer 
and winter institutes and by providing schools with leadership coaches. Intermediary B 
offered leadership retreats through its leadership development program. All 
Intermediary C principals attended an annual two-day leadership retreat in the summer. 
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Policy Mediation 



Scholars describe the ways in which intermediaries "exist between" individual schools 
and broader policymaking organizations, helping schools to mediate or navigate 
external policy conditions and relationships with district offices (Honig & Hatch, 2004) 
The principals of all six case- study schools described how their intermediaries helped 
them to navigate New York City Department of Education ( DOE) policy requirements 
and to negotiate with specific DOE offices. 



Honig &Hatch (2004) describe the ways in which 
intermediaries "buffer" the negative effects of 
policies that are not coherent with the work of 
schools. In interviews, principals described times 
when the intermediaries exerted influence with the 
DOE. For example, the principal of School 1 
described how the intermediary interceded with the 
DOE admissions office: "[Intermediary A] has been 
helpful in not allowing the DOE to overload us with 
incoming 9 th graders. This school is designed for 100 
students max per grade, and they tried to increase it 
to 108, but [Intermediary A] was able to have 
conversations with DOE to cut the number back to 
100 ." 



Bridging Policy Demands 

Intermediary C network leaders were concerned about 
the effect on Intermediary C schools of a new NYC DOE 
requirement to establish formative assessments, 
multiple- choice tests to be administered every six to 
eight weeks in all classes leading to a Regents exam. 
When they first heard of the impending requirement, 
network leaders worked with principals to design their 
own set of assessments and got the DOE to accept them 
in place of the city's standardized assessments. By 
bringing information back to the principals, the 
intermediary was able to "bridge external 
demands" (Honig & Hatch, 2004) and actually shaped 
the terms of compliance with the new policy. In an 
interview, the principal of School 6 said, "The 
formative assessment is the biggest undertaking that 
has affected our lives in a positive way. We were able 
to circumvent the DOE' s demands that we administer 
an additional set of standardized tests to our students. 
We were able to create a product, a set of rubrics that 
were aligned with our practices." 



The principal who took over School 4 when it was 
struggling described the way in which Intermediary 
B worked to improve relations with leaders in the 
community. In another example of Intermediary B's 
support, leaders at the intermediary exerted their 
personal influence to help individual students who 



were in trouble with New York City law enforcement. 

The principal related that, when two of his students were placed in jail overnight for a 
petty crime they likely did not commit, the principal contacted the head of the 
intermediary, who called the police and arranged for the youths' release. 



Both principals said that Intermediary C communicated with the DOE admissions office 
to ensure that it sent students who were appropriate for the schools, which serve newly 
arrived immigrants. In addition, the principals described the intermediary's extensive 
negotiations with the DOE focused on finding physical spaces near transportation lines 
for their new schools. 



Community Partnerships 

School- community partnerships marshal critical energy, resources, and support to help 
schools serve the social- emotional and academic needs of students (Honig et al., 2001). 
Some of these partnerships allow schools to expand learning into community settings. 
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providing youth with opportunities to use their newly emerging skills in cooperation 
with professionals (National Research Council & Institute of Medicine, 2004). In 
interviews and focus groups, principals and staff at schools supported by Intermediary B 
and Intermediary C noted that the intermediaries had helped the school foster 
partnerships with local organizations. These organizations provided supports for 
teachers, such as professional development opportunities, as well as student services 
such as college prep activities, social- emotional support, and experiential learning 
opportunities. 

Intermediary B's model of school support was to work primarily through one or more 
lead partners: community organizations that worked directly with the schools to provide 
professional development and develop collaborative instructional projects. At School 3, 
Intermediary B established a partnership with a local nature- studies organization that 
provided a full- year experiential science class for 9 th graders, professional development 
opportunities for teachers, and internships for students. In an interview, the principal 
explained that this lead partner played an active role in shaping the school's identity. 

She said that "all partners were involved with the curriculum development, professional 
development, school policy setting, and governance required to maintain the [science- 
oriented] theme." In follow-up interviews, teachers from this school described how the 
community partner helped them to create innovative inquiry- based projects. 

Teachers at School 3 also spoke positively about the support provided by the lead 
partner. In a focus group at School 3, one history teacher reflected on her work with the 
partner organization. She described a gardening project, in which students grew 
vegetables that were served in the school cafeteria, as "a very important supplemental 
piece of my teaching." In a follow-up interview, a science teacher at the same school 
described how he worked with this partner organization to develop inquiry- based 
curriculum. 

While we found that partner organization support helped some teachers to develop 
innovative practices and to improve their skills, we also heard from several teachers that 
partner support was not easily accessed or integrated into the overall work of the school. 
For example, a teacher in a focus group at School 3 said, "There needs to be more of a 
structure in place to get help from the partners— especially for other [non- science] 
departments. There's really no structure to get help other than to take the initiative to 
talk to them." Findings suggest that, while the partners provided high-quality support, 
this support was not always aligned with all teachers' needs and did not necessarily 
encourage teacher collaboration or leadership development. 

In an interview, the principal of School 4 described how Intermediary B helped his 
school establish a partnership with an outside organization that provided college 
support activities to students including an online application tracking program, college 
trips, college speakers, and a summer institute at an Ivy League university. In 
interviews, teachers mentioned that this partner did much of the school's college 
preparation work. This organization also provided training to guidance counselors, as 
well as curriculum for a weekly college application course for seniors, according to the 
principal interview. The principal also described in his interview how Intermediary B 
supported his school to establish a prestigious theater group as one of the school's 
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partners. However, teachers did not report working with this partner beyond taking 
students to theater events. 

In focus groups at Schools 5 and 6, teachers mentioned their work with a newly 
established Intermediary C teacher training program designed in collaboration with a 
local university. A teacher at School 5 described this program: 

We train our own teachers. Student teachers come through an interview 
process. Two candidates are placed in every [Intermediary C] school. They are 
graduate students, and they spend the entire year interning in our school. It's 
great if you can get to train your own people. 

Intermediary C has also facilitated connections with other community organizations 
that provided additional supports for students. For the past three years. Intermediary C 
has held a fair for community- based organizations attended by 25- 55 organizations that 
offered information and services to interested schools. Through this fair. School 6 
initiated relationships with an embassy that provided afterschool native- language 
instruction to students and with an immigrant support group that worked with the 
school's parent association to provide information about issues related to immigration. 



Experts say that intermediary organizations are important vehicles for providing 
professional development (McDonald & Klein, 2003; Rogers, 2007). Specifically, 
intermediaries can be instrumental in fostering teachers' capacities to implement 
effective instructional practices and support student achievement (Ball & Cohen, 1999; 
Cohen &Hill, 2001; Darling- Hammond, 1993). The literature emphasizes the 
importance of building on teachers' expertise and practice rather than relying 
exclusively on external expertise (Cochran- Smith & Lytle, 1999; Hatch, Eiler, &Capitelli, 
2003, Rogers, 2007). 

Across our interviews and focus groups at case- study schools, we found evidence of a 
variety of ways in which intermediaries provided professional development to support 
teacher learning. These opportunities included large professional development events 
such as conferences as well as professional development days in which teachers from 
various schools came together to participate in workshops and grapple with 
complexities of practice. We also found evidence that intermediaries provided ongoing 
professional support through instructional coaching, conferences and workshops, inter- 
visitations, and teacher study groups. These forms of support were most likely to be 
dted by principals and staff at the Intermediary A and Intermediary C schools. 
Intermediary B, as noted earlier, used a different model of support, working primarily 
through one or more lead partners who worked directly with the schools. This difference 
was evident in our teacher interviews, where 13 out of 16 Intermediary B teachers 
indicated that they were not receiving direct support from the intermediary. 



Professional Development 
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Instructional coaches 



In the two case- study schools served by Intermediary A, teachers received support from 
instructional coaches who led in- school professional development sessions and 

supported individual teachers. In teacher follow- 
up interviews, five out of 16 teachers, two from 
School 1 and three from School 2, described the 
support they received from instructional coaches. 
In a focus group at School 1, one teacher said, 
"[Our coach] has been an integral part of our 
development as a school. She's been in my 
classroom; she's evaluated me. She's been 
invaluable in terms of helping us develop teachers 
for this school and developing an academic 
culture." The coaching model seemed to be 
effective in providing support and in maintaining 
connections between the intermediary and the 
schools. 

Intermediary B also provided instructional 
coaches to schools. In a focus group at School 3, 
two teachers mentioned their work with a local 
support facilitator. This facilitator, according to 
the principal interview, was an instructional coach hired by the intermediary to spend 
two days a week working with the school. One of the teachers explained, "[The coach] 
helps with the running of the school leadership team, the inquiry team, and committees 
that run the school." 

Intermediary C approached support to schools in a different way. Rather than using 
instructional coaches. Intermediary C engaged a network approach that encouraged 
teachers to share best practices across schools. 



Coach-led Teacher Learning 

During a weekly staff meeting at School 2, an Intermediary 
A coach directed a session for teachers and administrators 
focused on running a class as a workshop. The focus of this 
session was guided by teachers' expressed desire to learn 
more about teaching classes in this format. The coach 
showed video footage of a workshop- style class taught by a 
seasoned teacher from this school. During the viewing, 
teachers used a worksheet provided by the coach to reflect 
on the video and then shared reflections in small groups. 
They elaborated on strategies that they observed, such as 
using both audio and video to present content and drawing 
on students' out-of-school experiences. Overall, teachers 
appeared to be energized by this opportunity to learn 
about the workshop model, to observe a peer teacher, and 
to reflect collectively on ways to improve practice. 



Conferences and workshops 

At the time of our study, both Intermediary A and Intermediary C provided professional 
development to teachers through annual conferences and professional development 
days, according to the organizations' websites and teacher interviews. Both 
intermediaries offered four- day summer retreats in which teachers attended workshops 
related to various aspects of instruction. These events also involved special sessions 
focused on familiarizing new teachers with the intermediaries' core principles and 
beliefs. 

Intermediary A held both winter and summer institutes for teachers. At the summer 
institute, teachers focused on curriculum development; they also met in sessions with 
their school teams and instructional coaches to develop action plans for the year. 
Intermediary A teachers spoke highly of these professional development opportunities. 
Of the 16 interviewed teachers, 13 made specific mention of the value of the summer 
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institutes and other professional development events sponsored by the intermediary . In 
a teacher focus group, one School 2 teacher said, "The [Intermediary A] professional 
development provides the best practices and strategies we can come back and 
implement." 

Intermediary C offered fall and spring professional development days in which teachers 
led workshops on various aspects of classroom practice. We observed an Intermediary C 
Election Day professional development workshop conducted by a teacher who explored 
ways of supporting students in the college admissions process. In this workshop, 
teachers and guidance counselors analyzed examples of students' college essays and 
discussed essay- writing guidelines. In follow-up interviews, all 16 Intermediary C 
teachers indicated that they had attended— and in some cases led— workshops during 
intermediary- wide professional development events. In addition, 11 out of 16 teachers 
described how the content and skills of the lessons we observed were developed in 
recent network professional development workshops. 

Teacher follow-up interviews and focus groups suggested that Intermediary B provided 
limited professional development. The schools in our case study tended to receive 
professional development directly from their community partners; however, a few 
teachers described having attended Intermediary B summer retreats in the past. In a 
focus group at School 4, two teachers had attended such retreats. One teacher at School 
3 criticized the intermediary's professional development sessions in a focus group, 
suggesting that effective professional development should involve "teachers talking 
about their experiences with the same kids and the same obstacles." These and similar 
comments suggest that the support offered directly by the intermediary, at least in these 
few cases, was not aligned with teachers' needs and did not foster ongoing teacher 
collaboration. However, some teachers at School 3, as described above, took a more 
positive view of the professional development provided by the school's lead community 
partner. 



Table 3.3. Teacher Professional Development by Intermediary 


Percentage of teachers who participated in selected 
professional development at least once in the last 
year 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 


Participated in professional activities involving 
teachers across schools in your district 


74% 


59% 


80% 


71% 


Observed model lessons in other schools 


23% 


41% 


51% 


38% 


Participated in grade-level or content-area 
study groups 


81% 


73% 


91% 


81% 


Participated in networks with teachers outside 
your school or district 


64% 


63% 


84% 


70% 


Source: Teacher survey (n = 143) 
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Inter-visitations and teacher study groups 

Teachers from all of the case- study schools reported on the teacher survey that they had 
participated in various cross- school events, including professional activities and 
networks with teachers from other schools (see Table 3.3, previous page) . These 
network-wide activities provided forums for reflecting on practice and solving common 
problems. The survey did not ask specifically about intermediary- sponsored events, but 
it is likely that many of these professional development events were sponsored by the 
intermediaries. Intermediary C teachers reported the greatest cross- school participation 
in such events, followed by Intermediary A and Intermediary B. 

In addition to the previously described cross-school 
conferences attended by most of the Intermediary A and 
Intermediary C teachers we interviewed, many teachers 
also participated in observations of model lessons in 
other schools— particularly teachers in Intermediary C 
schools, where more than half said they had observed 
lessons in other schools in the last year. In an interview, 
a School 3 teacher described an inter- visitation event 
held at his school in which teachers from other 
Intermediary C schools observed an advisory class 
together and then attended a meeting to discuss 
strategies for developing their advisory programs. 

Cross- school collaboration and sharing was also 
facilitated by teacher study groups. A large proportion of 
Intermediary A and Intermediary C teachers had 
participated in such a group in the last year. ( Some of 
these may have been within-school rather than cross-school groups.) Intermediary C 
provided release time and funds for materials to study groups composed of teachers 
from various schools who came together to work on an area of common interest and 
need. 



Tools 

All three intermediary organizations provided resources and tools to help 
administrators and teachers improve school and classroom practices. 

Intermediary A provided assessment tools designed to help teachers improve classroom 
practices. The intermediary worked in collaboration with a partner organization to 
develop a series of formative assessments used to measure growth in writing and 
mathematics between the 9 th and 10 th grades. It also developed periodic assessments to 
be used to fulfill NYC DOE testing mandates. These instruments were designed to help 
teachers assess student growth and to help schools identify areas for instructional focus 
and improvement. Additionally, the partner developed rubrics to assess implementation 
of the intermediary s core principles. Schools can use the rubrics to see how far along 



Inter- visitations and Study Groups 

Following an inter-visitation, a math teacher 
reported in his interview that he wrote a proposal 
through the intermediary's request for proposals 
(RFP) process to continue working with these 
Intermediary C math teachers in a study group. 
Through this RFP process, a teacher from School 6 
and two teachers from another Intermediary C 
school received funding to develop ten short-term 
proj ects on how to teach math through authentic 
and experiential projects. These projects were then 
uploaded to the intermediary's online curriculum- 
sharing system so that they could be disseminated 
to math teachers in other Intermediary C schools. 
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they are in implementing each principle and to guide their planning for school 
improvement. 

Intermediary B provided its schools with a student data system that allows 
administrators, teachers, and students to monitor student performance and progress 
toward graduation. It also provided school snapshot reports to principals and school 
staff at least twice a year. In addition, the intermediary supported schools in actually 
using the data, furnishing them with a data tools resource guide and sponsoring 
workshops for teachers on effective use of the data tools. Educators learned to use the 
data to understand student performance and to develop individualized plans to improve 
student achievement, while administrators learned to monitor school- wide performance 
and plan interventions. For further discussion of these data tools, see the next chapter 
on personalization and academic support. 

Intermediary C developed a series of resources as part of their effort to document and 
codify best instructional practices. Toolkits for new teachers in each discipline area 
include curriculum units and resources, reference articles, and textbook suggestions. 
Other resources include online videos of exemplary lessons and an online resource 
center offering curriculum guides and research papers; a planner tool that allows 
teachers to create, modify, and share lessons; and a teacher discussion board. In focus 
groups, teachers described how this kind of curriculum sharing was critical to helping 
them support their unique student population using Intermediary C's language 
development model. 



All three intermediary organizations in this case study emphasized personalization, 
academic rigor, and college preparation in their efforts to effect small high school 
reform. All provided a variety of supports such as policy mediation and professional 
development. Our case- study data suggest that the differences among the intermediaries 
are related not to their goals but to the means they used to achieve their ends. Specific 
emphases characterize each intermediary, particularly in regard to teacher professional 
development, which, as will be shown in Chapter 5, was a key factor in a school's ability 
to deliver academically rigorous instruction and effective college preparation. 

• I ntermediary A was distinguished by the use of instructional coaches to foster 
teacher and administrator professional development. 

• I ntermediary B relied on community partners to provide professional 
development opportunities as well as offering training in use of data analysis 
tools. 

• I ntermediary C used a network approach, connecting teachers and 
administrators to their counterparts in other schools supported by the 
intermediary so they could learn from one another. 

Differences in other areas appear also to be related to the models and structures 
cultivated in the case- study schools by their intermediaries. The rest of this report 
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examines how these models and structures played out in the schools' ability to foster 
personalization, provide effective instruction, and promote college readiness. 
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4. How Schools Foster 
Personalization & 
Academic Support 



The six case- study schools— indeed, all of the small schools funded by the Bill & Melinda 
Gates Foundation— serve high- need students who are at risk for poor educational 
outcomes. Most of these students are eligible for free or reduced- price lunch, many are 
special education students, and most come into high school with below- proficient scores 
in 8 th grade reading and math tests. ( See Table 2.2 on student demographic and 
academic data.) In addition, we know from interviews with teachers and focus groups 
with staff that students in the case- study schools face multiple challenges. They live in 
high- crime, high- poverty areas, with limited access to resources. Some are new 
immigrants to this country. Many live in single- parent homes or with a relative instead 
of a parent; these caretakers often hold several jobs. The students themselves often are 
responsible for younger siblings or for contributing to family income by holding a job. 

Schools that are effective in engaging urban students with high needs provide 
personalized instruction with specific academic and social- emotional supports in an 
environment where students feel supported and cared for (National Research Council & 
Institute of Medicine, 2004) . We examined the following components of such 
environments in the case- study schools: 

• Personalization that allows students to feel that teachers and staff care about 
their lives outside school 

• Academic support, particularly for struggling students 



• Use of data to promote both personalization and academic support 

• Home- school relations to involve parents and caregivers in students' academic 
progress 

Personalization: "They Know Us 
by Name Here" 

Small schools can, by nature of their size, offer students a personalized environment. 
Data from the teacher survey show that teachers fostered personal relationships with 
their students through frequent, individualized attention and interactions. For example, 
teachers from all case- study schools reported frequent interactions with students to 
discuss their interests and personal issues (see Table 4. 1). 



Table 4.1. Teacher-Student Interactions 

%of teachers responding they did the following "1^1 times a week" or "Daily" 


n= 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 

140-142 


Sch 1 

25-26 


Sch 2 

21 


Sch 3 

29 


Sch 4 

21-22 


Sch 5 

25-26 


Sch 6 

19 


Talk with students before or after 
class about their general interests 


65% 


81% 


62% 


62% 


81% 


79% 


71% 


Talk with students before or after 
class about personal issues in their 
lives 


40% 


62% 


59% 


59% 


76% 


58% 


59% 


Source: Teacher survey 



Indeed, students in focus groups said that they received a great deal of individual 
attention from teachers and staff and that they felt they could turn to these caring adults 
for assistance with personal or academic issues. 

They [teachers] develop relationships with students. When I feel like I have a 
problem, I feel like I can talk to someone. It's so nice that I can reach the 
principal at any time, and the teachers are very helpful. Teachers really care 
about the problems you have. They are willing to stay after school to teach us if 
we don't understand something or we need more help. ( 11 th grader. School 6) 

They have patience. They aren't just our teachers; they are our friends. They sit 
down and talk to you about personal problems, our educational problems, our 
problems outside of school, and they help us. (10 th grader. School 5) 

Students mentioned in focus groups that teachers and staff knew them by name and that 
they received personal, often one-on-one, attention. They felt they would not "get lost." 
Many noted that they felt they could turn to at least one, if not several, caring adults for 
assistance. 
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It's cool to be in a small school. Teachers are more "on us." They help us one on 
one. Big schools don't have time for one person. ( 11 th grader. School 1) 

I like it because it is small: one- on- one. Teachers try to make sure you learn. 

( 12 th grader. School 3) 

Students noted that they received support and personalization not only 
from teachers, but also from social workers, guidance staff, parent 
coordinators, and principals. As a 9 th grader from School 1 said in a 
focus group, "If the counselor knows you have a problem, shell grab 
you and the person you have a problem with and she'll make us talk, in 
order to try to prevent a problem. " 

This distribution of responsibility for students was corroborated by 
support staff in focus groups. For example, the parent coordinator at 
School 6 discussed the ways in which he connects with and supports students: 

With individual students, my door is always open. I work with students on 
college essays and [give them] a space to talk I try to help them to understand 
that their ideas are good. I am there as an individual tutor and support, and I 
like that role. They call me on my cell phone and email me. . . "Can you read my 
essay?" With some students, I call them early in the morning and try to get them 
here. 

This staff member illustrates the way in which staff in small schools 
often wear multiple hats: student cheerleader, tutor, academic advisor, 
attendance supporter, and more. 

Creating a personalized, caring environment is not without its 
challenges. An administrator at School 5, where students are new 
immigrants, commented in a focus group on the difficulty inherent in 
playing multiple roles to address students' complex situations and meet 
their needs: 

It is not a conflict so much but more the reality of working in urban schools 
where you have to wear so many hats. It's not a conflict, but reality. It's a part of 
working with young people. Sometimes, I'm the mentor, the mother, the 
advisor. 

The literature and research on best practices point to several structures that high 
schools employ to foster adult- student connections and personalized attention (Kemple, 
Herlihy, & Smith, 2005; Quint, 2008), including advisory groups and teacher/ student 
teams, referred to as instructional teams in two case study schools. Table 4.2 shows the 



"We hire teachers who are naturally 
interested in supporting immigrant 
kids, who understand that these 
students need more support than 
most. The kids know that their 
teachers will take the time to listen 
to them. "—Principal, School 6 
(interview) 



"I've heard stories from the bigger 
schools, and I wouldn't want to be 
there. They know us by name here. 
They are able to give you second 
chances here. You don't get lost as a 
student." —12th grader, School 4 
(focus group) 



Table 4.2. Organizational Structures That Foster Personalization 




Intermediary A 


Intermediary B 


Intermediary C 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Advisory groups 


/ 


/ 


/ 


/ 


/ 




Instructional teams 










/ 


/ 


Source: Principal survey 
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case study schools that implemented advisory groups and instructional teams. While the 
case- study schools used similar structures to support students and provide a 
personalized environment the features and processes characterizing those structures 
differed from school to school. 



Sample Advisory Group Structure 

At School 5, students in all four grades met in their advisory 
groups twice a week for 65 minutes each session. They had 
the same advisor for the first two years and then switched to 
another advisor in the 11 th grade. The curriculum focused on 
both the academic and emotional needs of students. The 9 th 
grade advisories used team-building activities to develop a 
cohesive group of students. Tenth graders started exploring 
their interests; in the 11 th grade, they started to connect 
those interests to possible career paths. In the 12 th grade, 
they looked for colleges that offer majors toward those 
career paths. Seniors were also involved in a program that 
matched them with outside mentors in the fields they 
wanted to pursue. 



College guidance counselors and advisory teachers worked 
jointly on college prep activities. Two students shared in a 
focus group that their advisor went the extra mile to help 
them get internships in their interest areas by driving one to 
the local hospital and the other to the police station. 



Advisory Groups 

Though the structures varied from school to 
school, advisory groups typically consisted of 
one staff member and a small group of students 
who stayed together at least for one school year 
and sometimes for all four. Advisory groups 
often served the dual purpose of providing both 
social- emotional and academic support for 
students. (Some schools also used advisory 
groups to support the college application 
process and preparation, as discussed in the 
chapter on College Preparation.) 



Five out of the six case- study schools reported 
implementing advisories in some form. At 
School 6, advisories were not in place in the 
2008- 09 school year, but had been in the past. 
At the end of the 2008- 09 school year, the 
school staff decided in a retreat that advisories 
were such a critical structure for supporting students that they voted to re- institute them 
( see Chapter 5 under Teacher Learning for a description of this retreat) . 



Table 4.3. Advisory Group Characteristics 




Intermediary A 


Intermediary B 


Intermediary C 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6* 


Grades served 


9-12 


9-12 


9 


9 and 12 


9-12 


9-12 


Nfeetings per week 


1 


1 


2-3 


grade 9: 2-3 
grade 12: 1 


2 


at least 1 


Curriculum 


Teacher 

developed 


Teacher 

developed 


No 

curriculum 


Grade 9: no set 
curriculum 
Grade 12: College 
Summit 


Set 

curriculum 


n/a 


Years advisor stays with 
same group of students 


4 


4 


1 


1 


2 


2 


^Advisory will be in place next year. 

Source: Principal and teacher interviews, teacher and support staff focus groups 
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The case- study schools implemented advisories in different ways. As shown in Table 4.3, 
advisories varied in terms of what grade levels were included, the number of years an 
advisor stayed with the same group of students, whether a set curriculum was used, and 
the number of times advisories met per week 



Instructional Teams 

Schools 5 and 6, following their 
intermediary's model, used "instructional 
teams" to provide a coherent structure for 
meeting students' academic and sorial- 
emotional needs. The teams consisted of six 
or so teachers who all taught the same 80- 
90 students. These teachers were 
responsible for providing not only academic 
but also social- emotional support for that 
group of students. At School 5, instructional 
teams met two to three times a week At least 
one of those meetings included a guidance 
counselor. The teams discussed the students 
who needed additional support: students 
who were at risk of failure, had sodal- 
emotional or behavioral issues, or were not 
working to their potential. The team also 
discussed students' noteworthy 
accomplishments, as well as students who 
might be overlooked because their issues 
were not acute or did not require immediate 
attention, ensuring that no one "slipped 
through the cracks. " A spedfic protocol 
guided these meetings (see box) . 

The prindpal at School 6 noted in the 
interview that the school's instrudional 
teams, which were organized similarly to 
those of School 5, were "critical in providing 
sodal and emotional support to students." 
Teachers used a "kid talk" protocol when 
they met with the guidance counselor and 
sodal worker to discuss individual students' 
needs and to plan for interventions. The "kid 
talk" protocol, which the prindpal had 
learned from another school in the 



Instructional Team in Action 

In the instructional team meeting we observed at School 5, 
teachers used two types of protocols to discuss how to support 
students. 

Bird's eye view. Teachers discussed 11 students who had issues 
including attendance, declining grades, depression, pregnancy, 
and others. Action plans involved a range of supports such as 
contacting parents, offering afterschool tutoring, counseling, 
and engaging fellow students for support. The plans were logged 
in a "blue book" to be reviewed and updated at each meeting. 

As one part of this segment of the meeting, the math teacher 
initiated a conversation about Maribel, who seemed depressed 
and had a number of absences. He reported that Maribel had 
said that "school doesn't matter" because she planned to move 
to Mfexico at the end of the year in order to escape her difficult 
living situation. After concluding that Maribel' s undocumented 
status was affecting her motivation, the team agreed to help 
Mkribel understand that a U. S. diploma would allow her to 
attend college in Mfexico. The English teacher said that he would 
ask a recent graduate who was attending college in Mfexico to 
speak with Maribel about the value of a U. S. diploma. 

In-depth descriptive review. The second half of the meeting 
focused on Samir, about whom teachers were concerned 
because his grades were falling. One teacher had prepared a 
written report that described Samir's strengths and weaknesses. 
Samir's four teachers then took turns describing first Samir's 
strengths and then his challenges. In the following open 
discussion, the teachers talked about engaging Samir through his 
strengths, including his passion for history and debate. The 
teachers noted that, though Samir struggled with reading 
comprehension, he didn't want to attend literacy classes 
because he perceived that they carried a social stigma. At the 
end of the discussion, two teachers agreed that they would 
follow up separately with Samir. The math teacher emphasized, 
"We need to be really careful not to be too negative with him 
because he is turning a comer, and if we are too hard on him, 
we might lose him." The team agreed that conversations 
emphasizing Samir' s strengths might convince him to take 
advantage of additional supports. Another teacher suggested 
returning to the "transcript review process" to help Samir 



1 Throughout this report, students' names are pseudonyms. 
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intermediary's network, solicits detailed observations of students' behavior without 
speculation about causes. At School 6 instructional team meetings, according to the 
principal, teachers in each content area made specific observations about a given 
student's social- emotional needs. Based on the observations, the teacher team and a 
guidance counselor or social worker made recommendations and assigned staff to follow 
up as needed. The observations, meeting discussions, and recommendations were 
recorded and kept on file. 



Personalization Structures That Work 



Data from our case study indicate that these six schools were successful in creating an 
environment where students received personal attention on social- emotional and 
academic matters— as indicated by the universal agreement in student focus groups that 
participants always had a caring adult to turn to when needed. The case- study schools 
varied in the extent to which those dedicated structures, such as advisoiy groups and 
instructional teams, were afforded dedicated time and space to focus on individuals as 




needed. Schools where the support for students seemed to 
be particularly strong had structures in place that allowed 
staff to meet frequently for the purpose of identifying 
student needs and developing appropriate support 
strategies. These structures fostered a coordinated staff 
effort, with teachers working with guidance staff, social 
workers, parent coordinators, and community liaisons as 
a regular operating procedure. What seems to set apart 
the two schools that had instructional teams is that this 
structure, implemented in frequent meetings, allowed 
staff to focus on individual students, using strong 
protocols that facilitated systematic reviews of all 
students' needs and progress. 



Fig. 4.1. Student 
support bulletin board 
at School 5 



The extent of personalized support available to students did not appear to be uniform 
across the case- study schools. Students from four of the six schools indicated in focus 
groups that they knew they could always turn to a teacher. In the other two schools, 
responses were more mixed. For example, students in all three focus groups conducted 
in School 2 generally agreed that some, but not all, of their teachers were available to 
help when they needed it. One 9 th grader said, "Some teachers are available when you 
need them and some are not." Similarly, students in a focus group in School 3 noted that 
teachers often were not available for extra help during the school day. 



Academic Support 

In addition to personal attention to students' social- emotional needs, staff at case-study 
schools provided individualized support for students' learning needs. In line with the 
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research on best practices (Kemple, Herlihy, & Smith, 2005; Quint, 2008), case-study 
schools also offered such academic support activities as tutoring, Saturday classes, study 
skills classes, and "catch-up" and credit recovery opportunities. 



Individualized Academic Support 

The extent to which teachers followed up on students' progress and gave them 
individualized attention is indicated in responses to several items from the teacher 
survey. More than half of surveyed teachers responded that they regularly gave personal 
attention to students' academic needs, as shown in Table 4.4. This support included 
reviewing attendance records and grades to identify students in need of intervention, 
assisting students with their work outside of class, and discussing students' progress 
with individual students and staff. There was a fair amount of variation on responses to 
these items by school. School 2 teachers more often reported giving various forms of 
personal attention, followed by Schools 3 and 6. 



Table 4.4. Personal Attention on Academics 












%of teachers responding they did the following "1^1 times a week" or "Daily" 












Intermediary A 


Intermediary B 


Intermediary C 






Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


TOTAL 


n = 


26 


21 


27-29 


20-22 


26 


19 


141-143 


Review attendance records to identify students 
with excessive absences from class 


39% 


62% 


62% 


73% 


42% 


47% 


54% 


Review grades to identify students who are 
falling behind academically 


50% 


67% 


67% 


59% 


38% 


32% 


53% 


Help individual students with their work outside 
of regular classroom time 


69% 


71% 


69% 


71% 


65% 


84% 


71% 


Follow-up on individual students' progress with 
other teachers, counselors, or administrators 


62% 


67% 


55% 


23% 


73% 


84% 


60% 


Talk with individual students about their 
progress in your class or other teachers' classes 


50% 


81% 


83% 


65% 


77% 


68% 


71% 


Source: Teacher survey 



Several students conveyed in focus groups that teachers cared not only about their 
social- emotional needs, but also about their learning, exerting extra effort to ensure that 
students learned. Students said that their teachers pushed them to work hard and 
succeed. Some even noted that students who did not succeed could not attribute their 
failure to teachers' lack of effort. As an 11 th grader said in a focus group at School 1, 
'Teachers are really on you, they push you to do your best. If people are failing here, 
they are doing it themselves. It's half up to you, and half them. If you put in the effort, 
they will put in effort too." 
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Responding to the question, "Where do you go for help with your schoolwork?" students 
in focus groups generally said that they go to their teachers, often reaching them outside 
of class by cell phone or e-mail: 

I go to my teachers. They make certain times to help students during the day. 
We can email them or even call them on their cell phones. Some of our teachers 
are just so busy during the day that the best way to get in contact with them is 
via email, and they always respond to us. ( 12 th grader. School 1) 

Usually you can find a teacher to help. Even after school, they are available. 
Teachers are always there— before school, after class, and during lunch. We can 
call teachers on their cell phones and email them. ( 12 th grader. School 5) 



Academic Support Structures 

The case- study schools used a variety of strategies to support students academically, as 
shown in Table 4. 5. All used extended periods and block scheduling for core subjects 
such as English and mathematics. All also offered afterschool tutoring, and all but one 
offered in- school tutoring. Afterschool tutoring was staffed by teachers in all schools. 
Schools 1 and 5 also used peer tutors; in School 5, the student tutors were paid, while in 
School 1, 11 th and 12 th graders earned extra credit for tutoring. 



Table 4.5. Academic Support Activities 




Intermediary A 


Intermediary B 


Intermediary C 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Extended periods/ block scheduling 


V 


V 


V 


V 


V 


V 


Study skills classes 


V 




V 


V 






In-school tutoring 


V 




V 


V 


V 


V 


Afterschool tutoring 


V 


V 


V 


V 


V 


V 


Saturday classes 


V 


V 




V 


V 


V 


"Catch-up" classes for students behind 
grade level 




V 


V 


V 


V 




Credit-recovery opportunities 


V 


V 


V 


V 


V 


V 


Partial rather than full credit for students 
who have completed some but not all 
course requirements 




V 






V 




Teacher advisor/ advocate assigned to 
individual students ("guardian teacher") 


V 


V 






V 




Source: Principal survey 



Support for failing or under-credited students 

The case- study schools used multiple strategies to support students who were failing 
courses or under- credited. All of the case- study schools offered credit- recovery 
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opportunities for students by allowing students to retake 
a failed dass for a new grade or to take summer school. 

According to the New York State Education Department, 
schools may also offer credit, if defidendes are in a 
defined area and re- taking the entire dass is not 
required, to students who make up those defidendes in, 
for example, afterschool tutoring sessions (Duncan- 
Poitier, 2009). Two schools also offered partial credit for 
courses in which students had completed some but not 
all requirements. The prindpal at School 4 emphasized 
that credit- recovery opportunities provided an avenue 
for meeting struggling students' needs. This school 
offered afterschool and weekend tutoring and courses, as 
well as summer school, for credit recovery. 

School 2 mandated afterschool classes for students who 
had failed a class; it also offered tutoring in English and 
math every day from 3:30 to 4:30 p.m. Students earned elective credits for attendingthe 
afterschool tutoring sessions. The principal noted in her interview that offering elective 
credit for tutoring sessions was an effective way to achieve high attendance. She also 
stated that the school's menu of afterschool and Saturday programs had "helped us to be 
on target for graduation." This school offered Regents preparation sessions on 
Saturdays, as did Schools 3 and 4. All six schools offered Regents preparation after 
school. 

While most of the feedback on afterschool assistance was positive, data from principals, 
teachers, and students at three schools suggest that it was not without challenges. Staff 
at School 6 observed that motivating students— particularly those who most needed 
support— to take advantage of the multiple opportunities for assistance remained a 
challenge. A teacher at this school said in a focus group, "When we offer afterschool 
homework help, students who really need the extra help are not taking advantage of it. 
We have tried everything from working with their schedules, to making it mandatory, to 
providing incentives, but we still struggle to get them to attend." 

In focus groups, students in two schools dted reasons for not taking advantage of 
afterschool and Saturday sessions. School 2 students said that teachers 
in particular subject areas were not always available during these 
sessions. Students in School 3 said that, though teachers from every 
subject were available at twice-weekly afterschool tutoring sessions 
and Saturday school, some students had difficulty attending these 
sessions because of responsibilities such as taking care of siblings or 
working. A few students also noted that the afterschool and Saturday 
sessions were not very helpful because they were crowded and noisy. 



" I don't think there is any way you 
can fall behind in this school. 
Teachers are really on top of things. 
They call home when they need to 
and they are always available for 
extra help during and after school. " 
—12 th grader, School 1 (focus group) 



Personal Academic Attention in a 
Collaborative Atmosphere 

At School 1, teachers monitored individual students 
in grade-level meetings that occurred two to three 
times per week. At the meeting we observed, the 
eight 10 th grade teachers discussed the academic 
and behavioral progress of three students who had 
been suspended. Some also discussed advisees they 
felt were not progressing. One teacher described 
calling an advisee's parent for help with getting 
the advisee to come to school. The parent in turn 
asked the teacher for help, revealing that she too 
was at a loss. The teacher asked her colleagues for 
suggestions she could bring to the parent. 
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"Guardian teachers" 



Schools 1, 2, and 5 assigned specific staff members to individual students as advocates 
and advisors. At School 2 , each teacher mentored one or two seniors who they believed 
needed particular assistance in their last year of school. One teacher said in a focus 
group that these mentors "are seen as [the student's] guardian teacher." The "guardian 

teachers" monitored students' credit 



Recognizing Students in a Grade-wide Meeting 

At School 1, we observed a 10 th grade "town hall" meeting, which 
started with announcements. The lead teacher announced that it 
was the end of the first semester— "which means college 
transcripts, " that is, emphasizing that performance counts 
because colleges look at grades. The teacher also encouraged 
students to participate in several upcoming charity drives, noting, 
"There are a number of activities in the school for people in 
need. " Students from one advisory group held a raffle for a 
wireless printer, the proceeds of which would go to a charity. 

The second part of the town hall was an award ceremony. Advisory 
teachers recognized students for "best achievement" and "most 
improved" in each core subject. Teachers also bestowed several 
"citizenship awards, based on student behavior in class and the 
halls." The lead teacher told students that the awardees were "a 
group of kids who were representative of who you should aspire 
to— how they conduct themselves in class, talking to each other 
and adults. These are kids who respect themselves and others, and 
they represent where you ought to be. " 

The last part of the town hall built camaraderie with a game 
called "Who's Smarter Than a Teacher?" Students had developed 
questions in the core subjects. Teachers picked a subject other 
than their own, and a student asked a question, for instance (in 
science), "How many calories (lower case c) are in a Calorie 
(upper case C)?" The teacher knew the correct answer, 1000. 
Students and teachers alike appeared to enjoy the game. 



accumulation to ensure they were taking 
all the classes they needed to graduate, 
encouraged students to attend afterschool 
and Saturday programs when needed, and 
provided one-on-one assistance with 
homework and study skills. "A lot of what 
we do is advise, more than anything else," 
said a teacher in a focus group. "They have 
low self- esteem, and we have to talk to 
them a lot. They eventually see themselves 
being able to make it [to graduation and 
college] ." This teacher also noted that the 
"guardian teacher" strategy 'has paid off," 
with more students passing Regents exams 
and certain classes. 



Incentives and recognition 
for academic achievement 

Most case- study schools used incentives to 
spur student achievement. The principal of 
School 2 said that students who passed all 
of their classes with a grade of at least 75% 
got a trip to Great Adventures amusement 
park. At least two other schools. Schools 3 



and 6, also used such incentives as pizza 
parties and field trips to reward good attendance or passing Regents exams. The 
principal at School 2 noted that the incentives had motivated many students, but that 



Table 4.6. Recognizing Students in Class 

%of teachers responding they did the following "\A times a week" or "Daily" 


n= 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 

142 


Sch 1 
26 


Sch 2 
21 


Sch 3 
29 


Sch 4 
22 


Sch 5 
26 


Sch 6 
18 


Recognize individual students in class 
for good attendance or good academic 
performance 


39% 


57% 


48% 


59% 


69% 


39% 


52% 


Source: Teacher survey 
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some students who were chronically absent or had failed too many classes still 
represented a challenge. 

Frequent recognition of student accomplishments is another important strategy to keep 
students motivated to work hard in school (Resnick & Flail, 2005). More than half of the 
teachers in case- study schools said on their surveys that they frequently recognized 
individual students in class for good attendance or 
academic performance, as shown in Table 4.6. 

Recognition occurred through other means as well. 

All of the case- study schools recognized student 
achievement with hallway displays including 
examples of high- quality student work and lists of 
students who made the honor roll, reached 
attendance goals, were accepted into college, and 
won citizenship awards. In School 1, students were 
recognized at least once every six- week grading 
period in grade- wide "town hall" meetings (see box 
on previous page) . 



Use of Data 

The research suggests that high-performing schools 
often use data to inform policy and program 
decisions, enhance classroom instruction and 
support strategies, and help students stay on track to 
graduate (Cobum &Talbert, 2006; Protheroe, 2001). 

Interviews, surveys, focus groups, and meeting 
observations revealed that the case- study schools 
used both quantitative and qualitative data in 
various ways to support student learning, inform 
policy and curriculum decisions, and assess student 
progress. 

For example. School 5, in a professional 
development committee meeting, used student 
portfolios to assess the extent to which 11 th and 12 th 
graders were ready for college. This review led the 
committee to decide to combine the "curriculum 
share" process, in which teams across the disciplines shared curriculum specifically in 
order to examine language development, with a review of student portfolios so that all 
instructional teams could look for ways to improve curriculum and instruction. 

The section above on instructional teams describes how two schools used teacher and 
staff observation and reports on students in their review process to assess students' 
needs and develop appropriate interventions. Another school. School 2, used attendance 



Examining Student Work to Inform Policy 
and Practice 

In a professional development committee meeting at 
School 5, teachers in various subjects and grade levels 
brought in examples of the work of five struggling 
language learners and discussed how to focus on 
language development and literacy in the content areas 
to meet such students' needs. One teacher commented 
that, though looking at student work with a language 
development focus was useful, she would prefer to look 
at portfolios more broadly to see how students were 
learning. The principal agreed that the group should 
focus on "how to create rich, layered curriculum that 
will really engage students." Consensus emerged that it 
would be useful for the committee to analyze portfolios 
of student work spanning all four years in order to 
evaluate how curriculum was meeting student needs. 
One teacher asked, "Can we look at a student's 
academic progression through their portfolios over 
three years and see how standards ratchet up over 
time?" 

The topic of college readiness came up as teachers 
focused on the importance of examining 11 th and 12 th 
grade portfolios. A teacher said, "We need to really 
focus on what is happening in the 11 th grade year 
because it is such a critical year. I wonder if there is a 
big jump in what we are suddenly expecting in the 12 th 
grade, and we are not providing enough scaffolding 
leading up to that. It would be good to look at these 
transitions through student work. " Other teachers 
expressed enthusiasm for this idea, suggesting that 
using portfolios would allow teachers to see student 
work in a variety of disciplines. 
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data in a weekly meeting to identify chronic absentees and students at risk of being 
chronically absent and to develop strategies for improving attendance. (See also Home- 
School Relations below.) At School 5, we saw a professional development committee 
examining student work to assess language learning and discussing a need to analyze 
student portfolios to see the progression of student work over the course of several 
grades and to think about how to prepare students for college ( see box) . Similarly, a staff 
retreat at School 6 began with student case studies to inform the group's assessment of 
and planning for student supports. 

We also observed teachers using student work to inform practice in an intermediary's 

annual fall network meeting. In this two- hour 
workshop on college- essay writing, participants 
worked in pairs or groups of three to review sample 
student essays, discussing the essays' strengths and 
weaknesses and the writing techniques students used 
to enliven their stories. Afterward, the whole group 
shared not only their impressions of the student essays 
but also suggestions for curriculum strategies that 
could be implemented in classrooms or guidance 
sessions. 

Schools 3 and 4 used sophisticated student data tools, 
developed and produced by Intermediary B, to assess 
student progress, inform curricular and instructional 
strategies, and keep students and parents informed about students' progress. The 
principal at School 3 described in an interview how her staff used "data snapshots" 
provided by the intermediary at least twice a year. The snapshots— graphic reports of 
Regents pass rates, credit accumulation, grades, graduation rates, and attendance— 
could be used to, for example, compare pass rates in individual classes with Regents 
pass rates. The principal noted that the snapshots allowed her to "see clearly which 
students are on track with credits and which are not." She further asserted that 
reviewing student data not only affected student achievement but also "allows for and 
enables reflection and encourages serious planning" (see box) . 

The principal of School 4 used the student data provided by the intermediary in a very 
different way. He posted color- coded displays— green for "on track" to graduate, yellow 
for "almost on track," and red for "off track'^n his office so that all students and staff 
could see where individual students and the school as a whole stood. Providing the data 
directly to students, the principal said in his interview, "empowers the kids to 
understand the data and their needs" and "motivates students to get more green on their 
charts than their friends have." A school administrator echoed this sentiment in an 
interview: "Some students take this as a 'wake-up call' because they want their [data] to 

2 Inquiry teams were developed in 2007 by the New York City Department of Education under the 
Children First Initiative. Each school was expected to establish a small group of teachers and 
administrators charged using data to identify a change in instructional practice that will accelerate 
learning for a specific group of underperforming students. Based on what is learned from that 
experience, teams work with school staff to implement and monitor system- level change to benefit 
all students (NYC DOE, 2008). 



Using Data to Improve Student Achievement 

A review of student data provided by the 
intermediary led School 3 to focus on social studies 
as a weak area. Having used the data "to identify 
teacher and student needs and address them,'' the 
principal said, she assigned social studies teachers 
to be part of an inquiry team. The teachers offered 
assistance, such as afterschool tutoring, to the 
lowest-performing students in social studies. 
According to the principal, the pass rate for the 
social studies Regents exam increased to 85% 
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show more green rather than less." He explained that he used the charts extensively in 
student- parent conferences, especially with the seniors, because they provided a clear 
representation of exactly where students stood on graduation requirements. 

In addition to the student data. Intermediary B provided schools it supported with other 
data tools, including: 

• A measure showing students' progress towards college readiness indicated by 
credit accumulation and Regents test scores 

• A summary of on- track performance for the entire school and a student- level 
spreadsheet showing the performance of individual students 

• A spreadsheet of individual student data organized by student and by teacher, 
which can be further sorted by grade, cohort, test score, and credit accumulation 

The case- study schools thus relied, to different degrees, on data to assess student 
progress, address student issues, improve curricular and instructional strategies, and 
inform students and parents of students' progress. They used quantitative data, such as 
rates of student attendance, credit accumulation, and test passing, as well as qualitative 
data such as observations and reviews of student work. 

As Cobum and Talbert's (2006) description leads us to expect, we found that teachers 
valued various forms of data, particularly those, such as observations and personal 
knowledge of students, based on their own clinical judgment. Some teachers and staff 
indicated that using quantitative data without qualitative data was less informative than 
using them together. For example, a teacher on School 3's inquiry team described in a 
focus group how illuminating it was to conduct 'low- inference observations" of 
classrooms— observations that focus on objective factors such as teacher questioning 
and student responses, rather than on subjective evaluations that require inference or 
interpretation— to investigate factors related to the improvement in achievement of a 
particular group of students. This teacher noted that teachers initially resented using the 
student data tools to assess the school's accomplishments, so that he and other teachers 
welcomed opportunities to look at students in a more nuanced way "and dig a little 
deeper and figure out what made [the students] success stories." Another teacher in the 
focus group agreed: '1 am glad that people are taking a look at the cultures that exist in 
our school. I think this is a good way of looking at this. " The first teacher said that the 
inquiry team also planned to conduct focus groups with students to gather additional 
data. 

However, using data was not always in and of itself a successful strategy for informing 
practice. At the same school described above ( School 3), we observed a department 
meeting in which teachers attempted to use data to inform strategies and instruction but 
had limited success in developing strategies that addressed the specific problem at hand. 
Teachers were given data sheets showing the passing rates for students in their own 
classes. The facilitator asked teachers to devise improvement strategies that would help 
more students pass their course. The discussion we observed did not lead to any new 
strategies to improve the passing rates. One teacher expressed frustration with the 
process of looking at passing rates, saying that this "data stuff" was just the new buzz 
word. Although this may be an idiosyncratic example, what we noticed that sets it apart 
from instances in which teachers' review of data led to specific changes in their practices 
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was that this meeting lacked a dear process for reviewing the data, diagnosing the 
problem, and developing strategies to address it. More spedfic strategies might have 
resulted if teachers had looked at actual student work or were able to analyze exactly 
why students were struggling and where the problems lay. Identifying a successful 
strategy is difficult to do without understanding the underlying cause of the problem. 
Spedfic and targeted information on the problem or issue, combined with a spedfic 
process for howto review the data and discuss appropriate strategies, such as the "kid 
talk" protocol used at School 6, may have yielded more spedfic suggestions on how 
teachers could support students. 



Home-School Relationships 

Home- school relationships are a critical component of effective schools. Parent 
involvement and high parent expedations about educational attainment have been 
linked to improved student performance, lower dropout rates, and increased post- 
secondary attainment (National Research Coundl &Institute of Medidne, 2004; 
Patrikakou, 2004)). Our study did not explore the extent to which parents were involved 
in the case- study schools or in their children's education; however, we did observe and 
document the ways in which these schools sought to involve parents and guardians. 
Spedfically, we found evidence that schools used a variety of methods to reach out to 
parents in three areas of concern: attendance problems, academic progress, and college 
preparation. Parent involvement in the college process is discussed in Chapter 6 on 
College Preparation. This section describes the ways schools worked to develop positive 
and supportive home- school relationships. 

In the principal interviews and staff and teacher focus groups, all of the case- study 
schools provided numerous examples of ways in which they reached out to parents and 
guardians in addition to typical parent- teacher conferences. According to the principal 
Fig. 4.2. Family survey, all six case- study schools had a policy of contacting the responsible adult if a 

association meeting student was failing a course. Five of the schools had a policy of contacting parents when 

poster at School 5 a student had more than two consecutive unexcused absences. 

Other ways in which case- study schools communicated with parents included: 

• Parent meetings and workshops, particularly ones related to college application 
and financial aid processes 

• Parent orientation meetings 

• Materials, again especially those related to college information, translated into 
several languages 

• Weekly newsletters from the principal 

• Reports of student progress through student data tools and EdLine classroom 
management tool 

All schools used more than one of these means of communication, though no one school 
used them all. 
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At School 1, at least some teachers kept parents informed using EdLine, an online grade 
book and classroom management tool where teachers can post students' assignments, 
attendance, and grades. Using a password- protected login, parents can directly view 
their student's progress. One teacher said in an interview that EdLine "gives the parents 
ownership. Parents can check and see how their children are doing. About one- half of 
the parents sign on to Edline. And for those who are signed on, the thing is to push them 
to keep looking at it and to have conversations with their child." 

In a focus group, another teacher said, "For parents who don't 
feel comfortable using the computer, I print out student [EdLine] 
reports for them and send them home for parents to review and 
sign off on." However, the system did not appear to be used 
consistently throughout the school. In a focus group, some 
students noted that some of their teachers did not update the 
system regularly with their progress data. 

At School 3, the principal sent a weekly newsletter to parents. 

The newsletter contained announcements about field trips, 
fundraisers, student progress reports having been mailed, and special events. It also 
included information about school policies on such issues as attendance, safety, and 
wearing school uniforms, as well as contact information for teachers and staff and 
notices of specific incidents such as the school scanner having broken down. 

The two schools serving students who are new immigrants also made special efforts to 
involve and inform parents. For example. School 6 translated the mission statement of 
the school into various languages and reviewed it with parents in orientation meetings 
at the beginning of the year. This principal also noted in the interview that staff at his 
school "spend a lot of time explaining to parents why this school looks different from the 
schools in their native countries" and assisting them with navigating the U.S. 
educational system. 

In our teacher survey, respondents reported contacting parents of 28 students, on 
average, during the 2008-09 school year (see Table 4.7). Variation among schools may 
be related to the diff erent, ways in which schools handled communication with parents. 
At Schools 5 and 6, contact with parents was conducted primarily through the students' 
instructional teams and the parent coordinator, so that not all teachers were responsible 
for parent communication. Rather, instructional team members took responsibility for 
contacting parents about attendance, academic, or social- emotional issues. For example. 



Informing Parents of Special Events 

One event of which the weekly newsletter 
at School 3 informed parents was a 
harvest day celebration. The lunch 
included a special salad grown "from seed 
to table" by students in the school's 
sustainable agriculture course. 



Table 4.7. Average Number of Parents Contacted by Classroom Teachers in 2008-09 




Intermediary A 


Intermediary B 


Intermediary C 


All schools 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Av. #of parents contacted 


31 


31 


29 


48 


15 


14 


28 


Source: Teacher survey (n=135) 



one instructional team described how they asked the parent coordinator to call a parent 
to discuss her son's need for additional tutoring. The student had signed a contract 
agreeing to go to in- school tutoring at least twice a week but had not been showing up. 
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The parent coordinator reported that he had spoken with a parent and hoped that, as a 
result the student would get some pressure from home to show up for tutoring. 

At school 2, attendance issues were dealt with through the attendance team: a group 
comprising the guidance counselors, the parent coordinator, and the community liaison. 
This group, we found in our observation of a meeting, took responsibility for contacting 
parents when students were absent three or more days in a row. 

Though we did not ask directly about parent involvement, teachers and support staff in 
focus groups conducted at schools 1, 2, and 3 noted that getting parents to attend the 
offered programs— for example, back-to-school night, orientations, and workshops on 
graduation requirements and college information— was a challenge. Teachers and 
support staff in focus groups conducted at Schools 1, 2, and 4 also noted that parent 
involvement in students' education was hindered by the difficult situations many 
parents faced, including holding multiple jobs and caretaking responsibilities. 



Summary 

Our case study data show that students in these small schools benefit from their 
personalized nature, with all of the students in our focus groups attesting that they knew 
at least one adult to whom they could turn to for help. Further, all case study schools 
had structures in place to foster personalization. However, some schools seemed to have 
more consistent, structured, and comprehensive academic and social- emotional 
supports in place for students. What differentiated schools with more comprehensive 
supports was that support structures (such as advisory groups or instructional teams) 
were: 



• Applied consistently to the entire student body, not only selected grades 

• Organized, with administrator support, to meet frequently and regularly 

• Characterized by use of both qualitative and quantitative data to assess student 
needs and inform decisions 

• Guided by structured protocols and procedures for identifying and addressing 
students' needs and ensuring that no student was overlooked 

While personalization and academic support systems for students are effective ways to 
address students' needs, they do not, in and of themselves, ensure that students attain 
the skills and content knowledge needed to be successful in college and life. Schools 
must also provide effective and academically rigorous instruction. In the next chapter, 
we describe the instructional practices observed in classrooms and described by 
teachers in the case- study schools. 
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5. Learning: How 
Schools Support 
Effective Instruction 



AED's case- study research is grounded in the most recent knowledge about practices 
that characterize instruction in high-performing schools (e.g., Allensworth & Easton, 
2007; Danielson, 2007; Edmonds, 1979; Goe, 2007; National Research Council & 
Institute of Medicine, 2004; Newmann, Bryk, &Nagaoka, 2001; Resnick&Hall, 2005; 
Tomlinson, 2000), especially schools that serve young people of color and students who 
are economically disadvantaged. We went into the case- study schools to document the 
extent to which instructional practices associated with student achievement were 
present, and to discern patterns that suggested facilitators and barriers to their 
implementation. All three intermediaries and all six schools in the case study valued 
effective, rigorous instruction. Our findings suggest that intermediaries and schools that 
focused on teacher learning were most effective in making this value a reality in the 
dassroom. 

To gain insight into instructional practices in the six Bill and Melinda Gates Foundation 
case- study schools, we observed eight classrooms in each school, for a total of 48 
observations. 1 We sampled a mix of grade levels and the four core subject areas: 
mathematics, English, sdence, and sodal studies (see Appendix B for a description of 

1 The dassroom observation protocol items and rubrics (see Appendix A) were adapted 

primarily from Danielson (2007) and Resnick and Hail (2005). Some items were developed by 
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the observed dasses) . The observations focused on documenting the extent to which we 
saw practices that the relevant research and literature assodate with improved student 
achievement. The observation data are supplemented by follow-up interviews conduded 
with the teachers of each of the observed dasses and by the survey of teachers in the six 
case- study schools. The interview allowed us to ask about, among other matters, the 
place of the observed lesson in the teacher's larger instructional plan and about pradices 
that happened not to be refleded in the observed lesson. The survey explored pradices 
employed by each teacher in one seleded dass. The limitations of these methods (see 
Chapter 2) means that the snapshot of dassroom pradices we captured cannot 
necessarily be generalized to the whole school. However, the spedfidty of the 
observation protocol, with its numeric ratings for various dassroom pradices (see 
Appendix A), allowed us to document the relative prevalence of particular instructional 
pradices, as was not possible with the other school practices we studied. 

We report below on the following aspects of pradice identified by the research and 
literature and explored in our dassroom observations, interviews, and surveys: 

• Effedive instructional technigues 

Clear purpose, instrudions, and explanations 

Hands-on instruction 

Modeling complex thinking and processing 

Building metacognitive skills 

Differentiated instrudion 

Freguent assessment and feedback 

Student grouping 

• Academic rigor 

Higher order thinking 
Quality guestioning technigues 

• Relevance 

Student choice in learning 
Personal and cultural engagement 
Real-world authentic tasks 

Combining the evidence from these three areas, we found effective, rigorous, and 
relevant instrudion in dassrooms in all six case- study schools. However, some schools 
more frequently employed strategies identified by the research as leading to student 
learning. Schools supported by Intermediary C most thoroughly incorporated effective 
instructional techniques in their dassrooms. When we looked for patterns to explain 
differences among schools, we saw that these schools offered strong support for teacher 
learning in ways that aligned with the research on effedive professional development 
that leads to effective instruction. The last sedion of this chapter, then, explores the 
supports schools and intermediaries offered to foster teacher learning. 
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Effective Instructional Techniques 



Central to promoting student learning-regardless of grade level or subject matter— are 
instructional practices that ensure students know what is expected of them, motivate 
and engage students in learning, address students' individual needs, and help students 
gain a deep understanding of the material and their own learning process (Danielson, 
2007; Resnick & Hall, 2005). Through observation of selected classrooms in the case- 
study schools, we have documented the extent to which we observed evidence- based 
practices that meet these goals, providing examples to illustrate what the practices 
looked like in classrooms. 



Clear Purpose, Instructions, and Explanations 



Using Danielson's framework for professional practice (2007), we observed and rated 
teachers' written or oral communication with students around their expectations for 
learning, directions and procedures, and explanations of content. These are key aspects 
of quality instruction, accordingto Danielson (2007), regardless of the grade level, topic, 
or content of the lesson. In our observations, teachers were rated in each of these 
domains on a four- point scale: Unsatisfactory, Basic, Proficient, and Distinguished (see 
Appendix A for the observation protocols, including scoring rubrics). Most of the 
teachers observed were rated as Proficient or Distinguished in all three categories, 
indicating that their communication of expectations, directions, and content was clear 
and appropriate for the situation and that the content was connected with students' 
knowledge and experiences. 

• 8 1% of the classrooms we observed showed Proficient or Distinguished levels of 
communication of expectations for learning 

• 92% showed Proficient or Distinguished levels of communication of directions 
and procedures 

• 86% showed Proficient or Distinguished levels of explanation of content 

These levels did not vaiy substantially by school, except that in School 3, only 38% (3 
out of 8) teachers were rated Proficient or above in communicating learning 
expectations. 

Aiding the communication about expectations and directions were written agendas and 
aims for the lesson. Over one- fourth ( 14) of the observed classes had written agendas or 
lesson aims, typically prominently displayed in the front of the room. One- third ( 16) of 
the observed lessons began with a "do now" exercise. These were short exercises, often 
lasting no more than five minutes, related to the lesson aims and objectives. For 
example, the "do now" exercise in a 12 th grade English class at School 3 had students 
sketch a set design for the play The Mousetrap. Teachers rated Proficient or above who 
did not communicate expectations and directions in writing provided explicit oral 
directions and expectations, clarifying as needed. 
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Hands-on Instruction 



Hands-on instruction has been shown to increase student engagement in class and to 
promote learning (National Research Council & Institute of Medicine, 2004) — 
particularly in science and math classes (Campbell, J olley, Hoey, &Perlman, 2002), but 

in other subjects as well. Teachers 

in the case- study schools reported 
in the survey that they used hands- 
on instruction fairly frequently, 
with nearly two- thirds (65%) 
reporting that they did so at least 
monthly and over one- third (37%) 
at least weekly. In our observations 
of classrooms, we saw lessons that 
included some form of hands-on 
instruction in nearly one- third ( 15) 
of the classes. About half (7) of the 
hands-on activities were observed 
in science classes, though they were 
also observed in math, English, and 
social studies classes. Examples of 
such instruction are provided in the 
box at left. 



Hands-on Instruction 

9-10 th grade global history: Used paper triangles to create tangram designs 
for a unit on Chinese history (School 5) 

9-1 0 th grade math: Used markers, string, ruler, scissors, softball, and 
paper to create a small package for a unit on measurement (School 6) 

9-10 th grade social studies: Used poster sheet and markers to "storyboard" 
a reading about King John and the Magna Carta for a unit on the feudal 
system (School 6) 

10 th grade math: Used Monopoly game houses in an exercise to find the 
mean, median, and mode in word problems (School 2) 

9-10 th grade earth science: Used plastic bottles, soil, rocks, and plant 
seeds to create "Ecosystems in a bottle— ESIB. " Students used rulers to 
measure growth of the plants in their ESIB, recording growth and other 
observations in a weekly journal (School 3) 



Modeling Complex Thinking and Processing 

Teachers model complex thinking by talking through and demonstrating the processes 
and steps they use to analyze and synthesize information and to solve problems. In 60% 
(29) of the observed classes, we saw evidence of teachers modeling complex thinking or 
processing for students. For example, a 10 th grade English teacher at School 3 verbalized 
her thought process on a reading- response assignment she had given: "Ask a question of 
your text and explain your thought process." The question the teacher asked of Catcher 
in the Rye was, "Will Holden ever be happy?" She explained, "My thought process was, I 
am wondering this because he seems totally depressed and has no goals or hope. " Later 
in the period, the teacher modeled inference making. As she read aloud from the text, 
she stopped to point out when she was making an inference: "I 'm going to model what 
inference is because we are working on finding quotes to support our statements: I'm 
going to infer that Holden is sweaty because he is nervous. . . I'm going to infer that 
Holden is good at heart; he gives the benefit of the doubt. You can point to these lines 
[in the book] as evidence." 

Another example of modeling was observed in an 11- 12 th grade math class at School 3. 
The teacher modeled howto solve algebraic equations by talking through his thought 
process in front of the class. This teacher also asked other students to model their 
though processes for solving problems. 
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Building Metacognitive Skills 

An important aspect of learning is developing and using what Resnick and Hall (2005) 
call self- monitoring and self- management strategies, or metacognitive skills. According 
to Resnick and Hall, metacognitive skills include noticing when one doesn't understand 
something and taking steps to remedy the 
situation, as well as formulating questions and 
inquiries that allow one to explore deep levels 
of meaning. Metacognition is also considered a 
necessary skill and behavior for success in 
college (Conley, 2007). Through observations, 
we were not able to assess the extent to which 
students used metacognition, which generally 
took place in students' thoughts. However, we 
were able to observe the ways in which teachers 
taught, modeled, and encouraged 
metacognition. 

Resnick (Institute for Learning, 2003) says that 
teachers can help students develop such skills 
by "modeling the ways that people notice and 
regulate their own learning processes," by 
pointing out when students have effectively 
used self- management strategies, and by 
making overt the thought processes that are 
usually carried out internally. Teachers can also 
comment on the quality of questions and 
responses in students' discussions and work 
(Institute for Learning, 2003) . Though we did 
not expect to see teachers providing instruction aimed at developing metacognitive skills 
in every lesson for every subject, we did see examples of teachers promoting 
metacognitive skills in nearly half (48%) of the observed classes. This instruction mostly 
took the form of teachers asking students to articulate the thought processes they used 
to solve a problem or form an opinion, which is also a form of modeling complex 
thinking, described above. 



Encouraging Metacognition 

Making internal thought processes overt: "How did you 
solve this equation?' "Does anyone else have another way to 
solve the equation?" (ll-12 th grade mathematics. School 3) 

Modeling/encouraging metacognitive behavior: "Check your 
notes. Do you think you took good notes on this topic? ... I see 
that some of us take notes during the video and some of us 
take notes later as we discuss with group-mates. It would be 
helpful to take notes as you go to help you remember. " (11 th 
grade English, School 5) 

Commenting on the quality of responses: The 9 th grade 
English teacher asked one group of students to listen to 
another group have a discussion about the merits of wearing 
school uniforms. The listeners were asked to write down 
comments or questions about the other group's discussion. 
After 10 minutes, the listener group asked questions and 
made comments about the discussion. The teacher 
encouraged the listeners to attend to the j ustifications and 
evidence students presented to support their opinion. She 
also commented on and asked students about the quality of 
the arguments. (School 2) 



Differentiated Instruction 

Differentiated instruction involves using a variety of instructional strategies to address 
diverse student learning needs. We saw many examples of teachers differentiating 
instruction in the classrooms we observed. Also, because differentiated instruction may 
occur without being visible to an observer, we asked teachers in the follow-up interview 
if and how they differentiated instruction in the observed lesson. Through observation 
and interviews, we determined that 83% (40) of the observed teachers differentiated 
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their instruction according to student needs. Teachers differentiate instruction by 
varying four dassroom elements (Tomlinson, 2000): 

• Differentiating the content refers to making adjustments to the skills or 
knowledge students are learning. In the case-study schools, we sawteachers 
allowing students to select books at different reading levels, using word walls 
and vocabulary exercises at diff erent, levels, adjusting the level of challenge of 
questions, and presenting information through multiple modalities: visual, 
auditory, and kinesthetic. 

• Differentiating process means making adjustments in the learning activities or 
strategies to fit students' needs. We saw examples in which teachers allowed 
students to choose topics of interest to them, used tiered activities such as a 
group project that had tasks with different levels of challenge for different 
students, provided students with extra supports such as graphic organizers, or 
allowed students extra time to complete work. 

• Products can be differentiated by varying their degree of difficulty. For 
example, some case- study teachers assigned different lengths of writing to 
students or allowed students to choose their roles in a group project. 

• Teachers differentiated the learning environment by using multiple forms of 
student grouping, such as pairs or small heterogeneous groups, and asking 
students to assist or tutor one another. 



Differentiating Classroom Elements 

Content. In a lesson at School 5 on algebraic equations with two unknowns, a 12 th grade math teacher presented 
four different methods (substitution, elimination, graphing, and matrices) to solve the equation. The methods 
were presented in multiple ways. The teacher created a guide that explained each method and provided sample 
problems. Students also had the option of watching a video that demonstrated one of the more advanced 
methods. Others were working on problems using the blackboard with teacher assistance. One student who had 
learning disabilities and was a visual learner was encouraged to use a number line to solve these problems. 

Process. In an 1 l-12 th grade physics class at School 1, all students were asked to solve for the maximum height of 
a football given its angle and velocity. Students were also to draw a picture of the situation described in the 
problem and to choose a coordinate system. Students who were more advanced, or who were done quickly, were 
asked to make an x/ y chart with all the data elements from the problem. If they finished that activity, students 
were instructed to choose an equation- solving problem from the advanced section of the textbook. 

Products. Students in a School 6 class were presented essay requirements for different colleges, ranging from Ivy 
League colleges to CUNY and SUNY schools. The teacher suggested that students interested in applying to 
Columbia attempt to write a 2000-word essay, while those interested in CUNY or SUNY write a two- to three-page 
essay. 

In a 12 th grade history class at School 1, students were allowed to choose how to present their final product: a 
report on a reform movement such as public education or prison reform. A group with strong writers and 
presenters decided to write a speech. A group with artistic talent decided to create a poster board. Another 
group decided on a combination of photos and writings. 

Learning environment. The 11 th grade chemistry teacher at School 2 grouped students strategically so that 
lower-skilled students were working with higher- skilled students. She also presented vocabulary definitions in 
multiple modalities while varying the content: orally, visually (on a handout), and kinesthetically, by having the 
students write the definitions down. 
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Frequent Assessment and Feedback 

One aspect of differentiated instruction is assessment. Teachers must frequently assess 
student progress to determine where students need assistance and which strategies are 
most effective for different students (Tomlinson, 2000). In nearly every class observed 
(44 out of 48), teachers used informal forms of assessment to gauge students' 
understanding and progress. Teachers used questioning and observation of student 
work to assess progress— for example, walking around the class or from group to group 
to look at students' work and listen to their conversations. Ten out 
of 48 teachers collected class work, such as "do now" exercises, 
and one teacher collected an "exit slip" to assess student learning 
(see box) . This social studies teacher said in her interview that she 
ends each class with such an exit slip exercise. In three classes, in 
addition to questioning and observing, teachers used more formal 
means of assessment such as brief quizzes or tests. 



"What new skills or knowledge did you 
learn in class today? Please write at 
least 3 complete sentences." —11 th 
grade social studies exit slip assignment 
(School 2) 



Student Grouping 

Effective instruction requires a variety of student groupings, depending on the purpose 
and format of the activity (Danielson, 2007). Nearly all of the observed lessons included 
more than one grouping method during a single class period. Teachers had students 
work individually, in small groups of two or more, or as a whole dass, whether in a 
teacher- led discussion or a teacher lecture. Only 13% (6) of the 48 lessons we observed 
used a single type of grouping throughout the entire period. On the survey, teachers 
reported using a variety of grouping practices; see Table 5. 1. One grouping practice that 
has been shown to foster higher achievement is peer collaboration (National Research 
Council & Institute of Medidne, 2004). Small-group work and peer tutoring are 



Table 5.1. Classroom Grouping Practices 

Teachers reporting they used the following grouping practices "at least weekly" 




Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


n = 


26 


20-21 


27-28 


21-22 


25-26 


17-18 


138-140 


Students work together in pairs or 
small groups on cooperative tasks 


77% 


95% 


68% 


77% 


96% 


95% 


84 % 


Students tutor other students 


58% 


60% 


39% 


62% 


84% 


83% 


63 % 


Students participate in teacher-led 
discussions 


65% 


67% 


89% 


77% 


62% 


56% 


70 % 


Students work alone guietly on a test 
or assignment 


69% 


43% 


68% 


64% 


38% 


41% 


55 % 


Students listen to teacher lecture 


65% 


85% 


75% 


68% 


27% 


47% 


61 % 


Source: Teacher survey 
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indicators of peer collaboration. Surveyed teachers reported using these practices 
frequently, with most teachers doing so at least weekly. 

Teachers in Schools 5 and 6, which are supported by Intermediary C, reported that their 
students often worked cooperatively and were generally less likely than students in other 
schools to work alone or listen to lectures. This finding is likely related to the 
intermediary's educational philosophy of emphasizing verbal interaction as a means to 
facilitate language development in its students, recent immigrants who tend to be 
English language learners. 



Academic Rigor 

Research studies on high-quality classroom instruction suggest that several factors are 
necessary for producing well- prepared students with high achievement outcomes 
(Danielson, 2007; Newmann & Associates, 1996; Resnick & Hall, 2005). The definition 
of academically rigorous instruction includes these two closely related kinds of 
classroom interaction that we studied in the case- study schools: 

• Higher order thinking: moving students' thinking to continually higher 
levels of abstraction 

• Quality guestioning techniques: inspiring student responses that not only 
use accurate and relevant prior knowledge but also require creativity and 
invention 

These characteristics of academically rigorous instruction can be observed directly. We 
designed the case- study classroom observation instrument to rate the occurrences of 
higher order thinking and high- quality questioning techniques. These 
components were then combined to develop a measure of academic 
rigor. In addition to the 48 classroom observations, we also examined 
interview, survey, and focus group data for mention of academic rigor or 
its components. 

Other characteristics that comprise the definition of academic rigor 
include those instructional practices that, over time, give students a 
solid foundation of core knowledge in the content areas (Institute for 
Learning, 2003) . These characteristics of rigor were outside the scope of 
this study because we conducted not multiple observations of individual 
teachers over time but rather observations of a cross-section of 
individual lessons. 



Higher Order Thinking 

To build higher order thinking skills. Resnick and Hall (2005) recommend that, with 
teacher guidance, students should hone their thinking abilities, construct explanations, 
and test their own understanding. They recommend that these higher order thinking 




Fig. 5.1. Bridge- 
building activity at 
School 5 
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skills "should be the daily fare of all students" and should be infused throughout the 
curriculum (Resnick & Hall, 2005, p. 18). Our approach in the case-study schools was to 
observe classrooms in all four major subject areas in order to detect the extent to which 
higher order thinking was present throughout the curriculum. 



Higher Order Thinking 
Highest rating 

A 12 th grade physics lesson at School 5 demonstrated the highest rating (3) of instruction 
that encourages higher order thinking as well as other effective instructional practices. In 
this hands-on lesson that integrated math and science, students constructed models of 
truss bridges and calculated their bridges' load capacity. Following the scientific method, 
students formulated hypotheses about the relationship between the number of triangles 
and strength of a bridge. Students worked in small groups of four or five, carrying on 
highly engaged discussions while the teacher provided encouragement. 

Students used building sets to construct three different types of truss bridges. They tested 
their bridge's strength by hanging books from different points in increasing weights until 
the bridge collapsed. Students then calculated the point of collapse, counted the number 
of triangles in the bridge, and recorded their findings. 

Students selected their roles in the group activity: Some built the bridges, others 
calculated and tested the load capacity, and others wrote the findings. The teacher 
incorporated differentiated learning by having students with higher math skills calculate 
the load capacity using a formula, while other students used a chart to guide them 
through the steps of the calculation. As the students worked, the teacher circulated, 
asking questions to encourage reflection. When one bridge broke, everyone laughed, and 
the teacher asked, "So what happened? Why did it break?" A student responded in 
Spanish. The teacher answered, "Good, so how many books did it take to exceed the load 
capacity?" The teacher continuously probed and assessed students' progress throughout 
the lesson: "What does this mean? How are you calculating the weight? What can you 
conclude from these data?" 

At the end of the class, the teacher asked students from each group to write their 
findings on a chart on the chalkboard. When the class found that two groups had recorded 
very different findings for one bridge, the teacher asked students to decide collectively 
which was the correct answer. Reviewing the scientific method, students spent several 
minutes hypothesizing and determining results. Ultimately they concluded that one group 
had made a procedural error. Then the teacher asked, "So what can we conclude from 
this data? A student answered, "More triangles, more capacity." The teacher followed up, 
"Can you answer in a complete sentence?" Laughing, the student said, "When you make a 
truss bridge, the more triangles you use, the stronger your bridge will be. " 

Lowest rating 

An example of the lowest rating (0) for higher order thinking came from an ll-12 th grade 
social studies class. The aim of the lesson was to identify the role of the President. The 
teacher first had students copy four vocabulary phrases: State of the Union address, 
diplomacy, foreign policy, presidential succession. He read these words aloud and 
presented a slide show of definitions and explanations as the students took notes. Then 
the teacher gave a handout to the students. It was labeled as a "reteaching activity" and 
seemed to be taken from the teacher' s workbook. Students looked up answers to 
worksheet questions in their textbooks. At no time were students asked to explain the 
reasons for their responses. The lesson included no discussion. 
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Newmann & Associates ( 1996) define higher order thinking as that which involves 
“manipulating information and ideas by synthesizing, generalizing, explaining, 
hypothesizing or arriving at conclusions that produce new meaning and 
understanding"' (p. 33) . Lezotte ( 1991) stresses the importance of teachers' striking an 
"appropriate balance between higher- level learning and those 
more basic skills" that undergird mastery at the higher levels (p. 
4). Lezotte also notes that "substantial staff development" is 
reguired to allow teachers to be responsive to the need for higher 
levels of learning. For the case- study schools, the balance 
between higher order thinking skills and basic skills defined a 
major challenge in educating the underprepared populations 
they served. 

In the classroom observations, the guality of higher order thinking in the instruction 
was rated on a four- point scale, with zero being the lowest rating, meaning that 
recitation of facts and formulas was seen rather than higher order thinking. The highest 
rating, 3, was for lessons in which the teacher acted as a facilitator while reguiring 
students to do the work of synthesizing, hypothesizing, explaining, drawing conclusions, 
making models, or producing new meaning and understanding. See the previous page 
for examples of the highest and lowest ratings on higher order thinking. 

We saw higher order thinking present to some degree in most (43 out of 48) observed 
lessons (see Table 5.2). Eleven lessons received the highest rating (3) in higher order 
thinking. An additional 24 classrooms received the next highest rating (2), indicating 
that the lesson freguently reguired students to apply information to new contexts, make 
judgments, and explain their reasoning. In 5 of the 48 lessons ( 10%), no higher order 
thinking was observed. 



Table 5.2. Higher Order Thinking 


n =48 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Number of observations ranked 3 or 2 (the two 
highest ratings out of 4) in higher order thinking 


6 


6 


4 


4 


8 


7 


35 



"There is an expectation of rigor in this 
school, and we try our best to meet the 
state obj ectives in terms of Regents 
scores. " —Science teacher, School 4 
(interview) 



Quality Questioning Techniques 

Danielson's (2007) framework for teaching identifies the quality of teacher questions as 
one component of rigorous instruction. Resnick and Hall (2005) also refer to 
questioning as an aspect of rigorous instruction. Students must be encouraged both to 
ask and to answer such challenging questions as, "Why do you think that? How did you 
arrive at that conclusion? What is your evidence?" Thus, questioning techniques that 
require students to dte evidence are also those that promote accountable talk, in which 
teachers require students to justify their arguments and responses, pressing for 
clarification and explanations when needed. 

In our classroom observations, questioning techniques were rated on a four- point scale: 
Unsatisfactory, Basic, Proficient, and Distinguished. Unsatisfactory questioning 
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techniques involved rapid-fire questions that posed little cognitive challenge and could 
be answered in one or two words. Distinguished questioning techniques, on the other 
hand, promoted thinking and encouraged students to make connections with other 
concepts or events and to arrive at new understandings of complex material. In a 
classroom with Distinguished questioning techniques, students themselves generated 
many of the questions. Examples of the highest and lowest ratings are given below. 



Quality Questioning Techniques 
Distinguished 

An 11 th grade social studies class at School 6 provided an example of the Distinguished 
rating for questioning techniques. To introduce this lesson on the Progressive Era, the 
teacher stressed that understanding vocabulary was key, asking students for synonyms of 
related vocabulary words. Later, students analyzed a political cartoon in which the "Lion 
Tamer" represented President Theodore Roosevelt. The teacher posed several questions 
about the cartoon to the whole class, sparking lively responses. 

"What might President Roosevelt's personality be like based on what you see in the 
cartoon?" 

"He looks serious." 

"He looks powerful." 

"He looks strict." 

"He looks fearless." 

"Does the cartoonist seem to believe that President Roosevelt will be able to control the 
trusts?' 

"He has a whip. I think so..." 

"Why do you think this?" 

"I don't know because the big businesses have the money and the power." 

"Yeah, but they have a monopoly and the businesses use their power over people 
who have no rights. " 

"The President will free the people from big business and create rights to protect 
the people. " 

The teacher frequently probed students and asked students to elaborate on their answers 
by providing specific examples. The responses elicited debate as to whether the President 
would be able to control the trusts or not. The teacher took this opportunity to explain 
the government's role in breaking up monopolies through "trust busting" to better serve 
the interests of the people. 

Unsatisfactory 

In an 10- 11 th grade science lesson, the teacher reviewed formulas and concepts students 
would need for an upcoming test by addressing a series of questions to the entire class. 
The questions required mostly one- or two-word answers and no further explanation. For 
example, the teacher asked, "If I wanted circular motion, where would the point of 
acceleration be?' Many hands went up, and students yelled out, "Center of the circle." 
The teacher immediately went on to another question. When one student yelled out part 
of the answer, the teacher finished the answer on his own without requiring more thought 
from the student. 
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Table 5.3. Quality of Questions 


n =48 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Number of observations in which questioning 
techniques were rated Distinguished or Proficient 


5 


5 


3 


5 


6 


7 


31 



Fifteen out of the 48 observations were rated Distinguished in terms of questioning 
techniques, and another 16 were rated Proficient the next highest rating (see Table 5.3) . 
A lesson was rated Proficient if most of the teacher's questions were of high quality and 
adequate time was provided for students to respond. 



Combined Findings on Academic Rigor 

We combined the ratings on higher order thinking and quality questioning techniques to 
rate each of the lessons as displaying high, medium, or low levels of academic rigor. We 
observed 30 lessons that received high ratings on both components of rigor, as shown in 
Table 5.4. We saw 10 lessons that scored in the medium range and eight that scored in 
the low range, showing little or no academic rigor. Most of the highest- rated lessons 
were in grades 11 and 12 ( 19). Observers rated more English classes highly in rigor than 
any other subject (10), followed by social studies (9), sdence(7), and finally math (4). 



Table 5.4. Instructional Rigor Ratings 


CO 

II 

PI 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Number of observations that scored high 
on both components of rigor 


5 


5 


3 


4 


6 


7 


30 


Number of observations that scored 
medium on both components of rigor 


3 


2 


1 


1 


2 


1 


10 


Number of observations that scored low 
on both components of rigor 


0 


1 


4 


3 


0 


0 


8 


Note: Each component was scored on a scale of 0-3, with the highest possible combined score being 6. A combined score of 4, 5, 
or 6 yielded a rating of high. Scores of 2 or 3 were rated medium, and scores of 0 or 1 were rated low. 



Lessons that earned high ratings on academic rigor tended to be characterized by: 

• Students working in groups to justify their opinions and responses, sometimes 
having to come to consensus 

• Students synthesizing ideas from previously learned materials 

• Teachers supporting students' analysis and thinking process by asking open- 
ended questions 

• Teachers introducing new ideas and encouraging students to ask questions 
about them 
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• Teachers suggesting a variety of resources where students could find evidence to 
develop their opinions 

Thus, a lesson that was rated high on academic rigor was one where the teacher 
supported students' work with open-ended questions that required higher order 
thinking. Once the students responded, they had to support the responses with evidence 
appropriate to the subject matter. 

Combining the components of rigor revealed eight lessons rated as low on academic 
rigor, as shown in Table 5.4. Lessons that earned the low rating tended to be 
characterized by: 

• Teachers presenting while students only listened or wrote 

• Students using formulas to figure out answers with little or no discussion of the 
work 

• Teachers and students giving one- word answers to questions 

• Teachers answering their own questions 

• Lessons that did not challenge the students to expand on what they already 



Not surprisingly, in schools where we saw more examples of lessons rated high on 
academic rigor, we saw few examples of lessons rated low on rigor. School 5, with seven 
lessons rated high, had no observed classes rated low in rigor. School 6 had six lessons 
rated high and none rated low. Schools 1 and 2 each had five lessons rated high and one 
or none rated low. Schools 3 and 4 had nearly the same number of lessons rated high 
and low, suggesting a great deal of variability in the quality of teaching in those schools. 

Teachers spoke in interviews of the ways in which they promoted academic rigor in their 
classrooms. They also clearly linked instructional rigor to college readiness, particularly 
to developing students' habits of mind (as discussed in Chapter 6 on College 
Preparation). For instance, a math teacher at School 5 said that teachers should allow 
students to assess for themselves howto approach problems. "They have to direct their 
own learning by analyzing and then synthesizing their knowledge on paper. This process 
encourages the kind of higher order thinking and reflection they will need to do in 
college." Similarly, a history teacher at School 3, when asked about preparing students 
for college, said that he focuses on analytical writing and discussion and on using 
evidence. Indeed, at every one of the case- study schools, at least some of the teachers 
interviewed referred to the components of rigor when asked how they help students 
prepare for college. 

The data indicate that, in each of the case- study schools, rigorous instruction was a 
priority. For example, an English teacher at School 1 said in a focus group: 

I want students to develop skills in expressing abstract concepts in writing and 
speaking, and that's very much college preparatory. In the English department 
[ 11 th and 12 th grades], we purposely choose sophisticated and challenging texts 
to read in the English dass, and students are expected to do very sophisticated 
writing. 

However, our data also show that, in the minds of teachers, factors such as time 
constraints and emphasis on preparation for Regents and other tests challenged the 
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delivery of rigorous instruction for many teachers. Prominent in the minds of faculty 
members of the case- study schools were worries about where the students actually were 
in their skill development compared to where they needed to be. Teachers were well 
aware that rigorous instruction is an important factor in college readiness, but they were 
equally aware that many students lacked some of the foundational skills and knowledge 
that would allow them to engage in more rigorous work. 

Preparing students for Regents exams and other tests can have the effect of pushing 
teachers to cover a broad amount of content superficially rather than focusing on depth 
of content and on the higher order thinking skills that prepare students for college. For 
example, a math teacher in School 3 lamented in the interview that he was not able to 
focus on the problem-solving skills he believed his students needed for the future. He 
noted that he felt "pressure to finish lessons on time to keep students on track towards 
Regents." He said that he'd like to do more hands-on exercises, but didn't feel he could 
spend the time. "That's the dilemma for teachers now. I 'm willing to follow the 
curriculum, but if I had the freedom. . ." 

In a social studies disciplinary team meeting we observed at School 6, four teachers were 
discussing students' ability to handle more rigorous curriculum. One teacher expressed 
frustration over the degree to which students were unable to move from scaffolding 
activities— in which students are provided supports and models that are gradually 
removed until the students can perform the task independently— to engage in higher 
order thinking. Other teachers, expressing concern about whether students were able to 
answer questions requiring higher order thinking, suggested stressing a small number of 
essential questions. Another teacher agreed that the emphasis should be on depth in a 
smaller set of concepts rather than on breadth, so that students could do higher order 
thinking in fewer areas. 

Students also expressed their perception of the tension between time and depth, 
speaking in focus groups of the extent to which they were challenged in class and the 
pace at which teachers moved through the curriculum. A group of 9 th graders from 
School 1 mentioned that teachers needed to be more patient and explain things more. 
One said, "They should not spend just one day on a lesson, and they should go into more 
detail and spend more time on things. They shouldn't only stick to big topics but go 
deeper into the subjects." Apparently these students agreed with the teachers who 
believed that depth of instruction might better suit the students' needs than breadth. 



Relevance 



Studies show that student engagement increases when topics are personally interesting 
and related to students' lives and when students have some input into what they learn 
and howthey learn it (National Research Council Institute of Medicine, 2004). Further, 
these studies emphasize the value of "authentic" tasks that have some meaning outside 
the world of school; such authentic tasks can include real-life experiences, experiential 
learning, and writing for audiences other than the teacher (Newman & Associates, 

1996) . Our research considered the extent to which classes in case- study schools: 
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• Provided opportunities for choice 

• Used personally and culturally engaging practices and materials 

• Provided opportunities for students to engage in real-world "authentic" tasks 

The findings below about these aspects of relevance are based primarily on classroom 
observations. Opportunities for student choice or use of real-world tasks might not take 
place in every lesson, so their absence does not necessarily signify weak instruction. 



Student Choice 

Studies reveal that involving students in decisions about what and how they learn leads 
to higher levels of achievement (American Institutes for Research & SRI International, 
2005; National Research Council &Institute of Medicine, 2004). As noted earlier, giving 
students choice is also one strategy teachers use to differentiate instruction. Indeed, 
many teachers we interviewed said that giving students choices was an intentional effort 
to meet different students' needs. In our observations of 48 classrooms in case- study 
schools, we saw instances in which students were given opportunities to choose such 
aspects as their roles in classroom projects, the particular content they studied, how 
they solved problems, and how they would present their learning. 

For instance, in a 12 th grade chemistry class at School 6, when students were creating 
chemistry lab videos to illustrate concepts of chemical mixtures, they chose which role 
they played in their groups: script writer, actor, director, or video producer (see box 
below under Effective Classroom Practices). In an English class at School 5, students 
conducted group research projects about various social reform movements. They were 
allowed to choose which content to research; for example, one group chose to research 
women's suffrage. Each group also decided what kind of product— for instance, poster, 
slide presentation, or speech— it would create to present what the group had learned. 
Providing students these options were also differentiation strategies, as described 
earlier. 



Table 5.5. Student Choice in Learning 


n =48 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Number of classroom observations in which students 
were allowed to choose what or how they learned 


1 


2 


1 


1 


5 


5 


14 



In our classroom observations, we saw evidence that students were given choices about 
how and what they learned in 29% of the 48 lessons observed. Table 5.5 shows 
considerable variation among schools in the extent to which we saw evidence for student 
choice in classrooms, with substantially more observed classes in Schools 5 and 6 
allowing opportunities for choice than classes in the other schools. 
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Personal and Cultural Engagement 

Studies show that students enjoy learning more— and learn better— when topics are 
personally interesting and related to their lives (National Research Council & Institute of 
Medicine, 2004). Further, research reveals that classroom practices that tap into 
students' prior cultural knowledge and that acknowledge and legitimize their cultural 
histories increase engagement (Gay, 2000; Ladson Billings, 1992). Such topics and 
practices enable students to contextualize and incorporate experiences across the 
contexts of home, community, and school (Doherty, Hilberg, Pinal, & Tharp, 2003). 

Our classroom observations showed evidence of practices that engaged students' diverse 
backgrounds. The two case- study schools that served immigrant students paid explicit 
attention to designing structures and practices that acknowledged the diverse 
backgrounds and experiences of their students. For example, in a science class at School 
5 and a social studies class at School 6, we saw students being encouraged to conduct 
research and to write in their native languages. Allowing students to 
work in their native language, according to Gutierrez (2002), signals 
respect for their diverse linguistic backgrounds. An English teacher at 
School 5 encouraged students to make connections between a video 
about the exploitation of immigrant workers and the economic 
conditions in their native countries. In response, one student in this 
class said, "It's kind of like the economy in my country. Our money is 
worthless, and you have to take a suitcase hill of money just to buy a 
loaf of bread." This teacher made several references to the native 
countries of the students, thereby acknowledging their diverse 
experiences. We saw teachers in these two schools making education 
relevant to these students by respecting their cultures and languages, 
providing opportunities for native language use, and making 
connections to students' backgrounds. 

In other schools, we observed instances in which teachers designed lessons focusing on 
issues that are socially relevant for urban students. For example, an English teacher at 
School 1 asked students to debate a series of statements related to gang membership in 
preparation for reading a novel about a gang leader in Los Angeles. In an English class 
at School 4 focused on developing writing skills, the teacher asked students to choose 
which of two rap stars was better, Tupac or Biggie, and then to provide evidence for their 
answers. Another English teacher at the same school described in her interview how she 
redesigned her English class to focus on books that are culturally and socially relevant 
for her students. In our observation of this teacher's class, students discussed a novel 
about a Sudanese family's move to the U.S. The teacher shared her immigration story 
and then asked students to discuss and write about their own immigration stories. 

Table 5.6 (next page) displays the number of classroom observations in which teachers 
made connections with students' cultures and personal lives. We saw this aspect of 
relevant instruction in 79% of the 48 observed classes. School 3 had notably fewer such 
classes than did the other case- study schools. 



" I want to thank the school for 
giving us the feeling that our culture 
is not bad.... Most of the time in 
American schools, they want you to 
forget about your culture; they are 
always putting down other cultures. 
It's really helping me to trust in 
myself and trust in my roots, and to 
take that to go forward, not to cut 
them off. " — Student from 
Bengladesh, School 5 (focus group) 
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Table 5.6. Connections with Students' Out-of-School Experiences 


n =48 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Number of classroom observations in which teachers 
made implicit or explicit connections with students' 
substantive knowledge and personal experiences or 
public problems 


8 


6 


3 


6 


7 


8 


38 



Real-world Authentic Tasks 

Research shows that instruction that allows students to learn in real-world settings is 
more engaging and results in higher academic achievement (Lee, Smith, & Croninger, 
1995). Newmann & Associates ( 1996) also emphasize the value of instruction that 
engages students to produce "discourse, products, and performances that have value or 
meaning beyond success in school." Several of the case- study schools had programs that 
provided opportunities for students to engage in inquiry in real-world settings. 



Internships 

In five of the six case- study schools, surveyed 
principals reported that they offered internship 
programs that provided students with 
experiential learning opportunities in 
community settings. The exception was School 
2. The internship program at Schools 5 and 6 is 
described in the box. The principals of Schools 1 
and 4 said on their survey that the schools 
offered internships, but we cannot describe 
them because they were not mentioned in 
interviews or focus groups. School 3 had 
established an internship program with its 

community partner, a nature- study organization. Eight students were involved in 
internships in which they worked with scientists on a study on seed flotation, according 
to the principal interview. Such work in the community with specialists enables students 
to produce new knowledge that has meaning beyond the walls of school (Newman & 
Associates, 1996) . Further, community internships engage students in authentic work 
that makes the real-world value of learning more evident. 



Internships in the " Real World" 

In Schools 5 and 6, all students were required to attend 
internships in the spring of their junior year. Students 
worked with their advisors to identify and apply for 
internships based on their professional interests. They then 
worked at their internship placements three afternoons a 
week for a total of 144 hours. Students were placed in 
diverse workplaces including law offices, schools, advertising 
agencies, and hospitals. The principal of School 6 explained 
in the interview, "Internship hits two of our core principles: 
experiential learning through exposure to the professional 
world of work and language development. It is a powerful 
tool that allows students to grow socially and emotionally 
and get a glimpse of what a post-college life might look like. " 



Community partnerships 



At School 3, the real-world learning fadlitated by the school's community partner did 
not end with the internship described above. In addition, all 9 th graders at this school 
attended a year-long field studies class taught by a science teacher from the school as 
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well as by scientists and educators from community partner organizations. These 
students spent one full day a week conducting natural experiments at the community 
partners' outdoor sites. In an interview, the principal exclaimed, "They conduct 
scientific inquiry. They live it!" In addition to their experiments, students participated in 
experiential learning expeditions in locations such as the Florida Everglades and upstate 
New York. One student even participated in research on polar bears in Manitoba, 
Canada. Students at this school also spent one afternoon a month in field trips with 
community partners as part of their community service requirement. 

In classroom observations at this school, we saw evidence that teachers attempted to 
integrate what students had learned in their field studies into classroom practices. For 
example, in one science dass, students created ecosystems in two-liter bottles. The 
teacher asked, "In your field studies, when you got up to Fallkill Falls, what did you 
find?" Students then brainstormed the components of an ecosystem. The teacher made 
the classroom learning more relevant to students by connecting it to the real sdentific 
inquiry they were doing in their field studies. 



Effective Classroom Practices 



When we move to the bigger picture to analyze use of effective dassroom pradices by 
school, we see variation in the frequency with which these pradices were implemented. 



Table 5.7. Frequency of Effective Practices in Observed Classrooms 

(number of classrooms out of 8) 






Intermediary A 


Intermediary B 


Intermediary C 




Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Clear purpose, instructions, and explanations* 


6.7 


8 


5 


6 


8 


7 


Hands-on instruction 


1 


3 


2 


0 


5 


4 


Modeling complex thinking & processing 


4 


5 


7 


3 


5 


6 


Building metacognitive skills 


4 


5 


3 


1 


6 


4 


Differentiated instruction 


4 


7 


7 


6 


8 


8 


Frequent assessment and feedback 


7 


8 


7 


6 


8 


8 


Peer collaboration / student grouping 


4 


6 


3 


1 


7 


7 


Higher order thinking 


6 


6 


4 


4 


8 


7 


Quality questioning techniques 


5 


5 


3 


5 


6 


7 


Student choice 


1 


2 


1 


1 


5 


5 


Connection to students' lives outside school 


3 


4 


2 


4 


3 


5 


Average frequency of effective practices 


4.2 


5.4 


4.0 


3.4 


6.3 


6.2 


Source: classroom observations 

*This item combines three separate items on the observation protocol that are all related to teacher 




communication. To calculate the school score, we averaged the number of classrooms from the individual items. 
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Snapshot of Rigorous, Effective Instruction 

A 12 th grade chemistry class we observed at School 6 was working on a unit whose goal 
was for students to gain a full understanding of types of mixtures. Working in groups of 
four or five, students were producing chemistry videos about homogeneous and 
heterogeneous mixtures. Following the scientific method, students had to develop a 
hypothesis, design an experiment to test it, explain what happened in their experiment, 
and draw conclusions— all recorded on the video. 

Students could choose which components of the project they wanted to work on: writing 
the video script, acting, directing, recording, or editing the video. Since each task 
emphasized key concepts, these multiple entry points were an effective way of 
differentiating instruction. Additionally, the tasks built on non-science skills such as 
writing, organizing and managing a proj ect, oral communication, and using computer 
video technology. 

Students were given choice in selection of materials for the mixture experiments and in 
the settings for their videos. For example, one group created a Crazy Chef skit in a mock 
restaurant kitchen. The ingredients— a cup each of hard candies, coffee, and vegetable 
oil— were displayed on a table. As the video rolled, the narrator presented a problem: 
"What will happen when our chef mixes these ingredients together?" The assistant chef 
stated her hypothesis, "They will maintain their properties and remain separate. Since oil 
is less dense than coffee and candies, it will rise to the top. " When the chef mixed the 
ingredients, describing each step, the assistant chef's hypothesis was borne out. The chef 
asked, "Do you want to taste my special cocktail?' The assistant chef turned up her nose. 
Then the narrator summarized the results of the experiment: "The candies, coffee, and 
oil maintained their individual chemical properties. The hard candies, as solids, stayed at 
the bottom of the container. The coffee and oil— both homogeneous liquids— turned into a 
heterogeneous mixture. So our Crazy Chef and his assistant have proven their hypothesis." 

The teacher supported the students in their work rather than directing it, promoting an 
atmosphere of respect and trust. In the period we observed, most groups were at the 
stage of editing their video. They were scattered around the classroom, in a separate 
room in the back, and in the hallway. Even when the teacher left the main classroom to 
check on other groups, the students remained on task. 

When asked how the lesson fit into the larger picture of preparing students for college, 
the teacher said, "Working collaboratively in groups is always a skill that we reinforce 
here at [School 6], and I believe working collaboratively is something very important and 
necessary for moving on to college. Also, the technology that students have access to and 
are learning will be great asset for them and will help them prepare for college. " 



as shown in Table 5.7. An example of effective classroom instruction from a case- study 
school appears above. 

We measured student engagement in classroom observations by rating to what extent 
students appeared to be on- task and interested during most of the lesson, on a scale of 1 
to 4 (see rubric in Appendix A) . Not surprisingly, student engagement, indicated by the 
top two ratings on our scale, was more frequently observed in schools where we also 
observed greater use of effective practices, as shown in Table 5.8 (next page) . 

These patterns strengthen our portrait of effective instruction in the case- study schools 
and show that some schools apparently had a higher incidence of high- quality 
instruction than others. Schools 5 and 6 both had similar levels of high frequency of 
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Table 5.8. Student Engagement 


n =48 


Intermediary A 


Intermediary B 


Intermediary C 


TOTAL 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Number of classroom observations in 
which most or all students demonstrated 
engagement by being on task 


7 


7 


5 


5 


8 


8 


40 



effective instructional practices, at 6.3 and 6.2, and students demonstrated engagement 
in every one of both schools' classroom observations. School 2 comes dose to this high 
standard on both measures. School 1, like School 2, had engaged students in seven of 
eight classroom observations, though its average frequency of effective practices was 
somewhat lower. Schools 3 and 4 had lower frequency of effective practices and less 
evidence of student engagement than the other schools. 



Teacher Learning and 
Instructional Coherence 



To explore fadlitating and constraining fadors to implementation of effective classroom 
pradices, we looked for structural, policy, and procedural fadors that may explain the 
differences by school. In the schools where we saw greater frequency of effedive 
pradices, we also saw greater levels of teacher collaboration and support for teacher 
learning, resulting in more instructional coherence. 

AED's research explored elements of teacher professional communities by looking at the 
extent to which teachers in the case- study schools were engaged in produdive 
professional development that furthered their capadties to deliver high-quality 
instruction. Scholars point to the ways in which teacher learning and reflection with 
peers supports the collective capadty of schools to design and implement effective 
pradices (Little, 1999; McLaughlin &Talbert, 2001). In the case-study schools, we saw 
considerable variation in the degree to which teacher learning was supported. 
Spedfically, in the schools were we saw the most frequent use of effective practices, we 
saw stronger supports for teacher learning, including: 

• Dedicated time and space for formal teacher collaboration 

• A strong culture of reflection and inquiry that guided such collaboration 

• Teacher participation in the design and implementation of professional 
development 

• Emphasis on distributed leadership 

Based on these four fadors, we used data from our interviews, focus groups, and surveys 
to assign each school a rating of strong, moderate, or limited support for teacher 
learning. The schools in which we found stronger evidence of support for teacher 
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learning were also those in which we found stronger evidence for effective instruction, 
as shown in Table 5.9. 



Table 5.9. Effective Instruction and School Support for Teacher Learning 




Intermediary A 


Intermediary B 


Intermediary C 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Average freguency of 
effective instructional 
practices* 


4.2 


5.4 


4.0 


3.4 


6.3 


6.2 


Support for teacher learning** 


Moderate / 
Strong 


Moderate / 
Strong 


Limited 


Limited 


Strong 


Strong 


*To calculate the school average frequency of effective practices, we averaged the number of classrooms where we 
observed the 11 practices listed in table 5.7. 

"^Ratings are AED researchers' judgment based on interview, focus group, and observation data. 



Schools 5 and 6 showed both the highest frequency of effective instructional practices 
and the strongest rating in support for teacher learning. The fact that these two schools 
were also characterized by instructional teams that provided an effective, structured 
means of supporting students may be related to these findings. Schools 5 and 6 were 
also among the four case- study schools characterized by an integrated, school- wide 
approach to college preparation, as described in the next chapter. Though we cannot 
prove that these factors are related, the research dted above (Little, 1999; McLaughlin & 
Talbert, 2001) leads us to expect that strong teacher collaboration would co-occur with 
strong personalization structures, strong college preparation features, and strong 
instructional practices. 



Teacher Collaboration 

While teachers at all of the case- study schools reported some degree of collaboration 
with their peers (see Table 5.10), our interviews and focus groups revealed sharp 
differences in the degree to which such collaboration was formalized using structures 
and routines, as well as in how teachers used their common planning time. 



Table 5.10. Frequency of Teacher Collaboration 


n =143 


Intermediary A 


Intermediary B 


Intermediary C 


Sch 1 


Sch 2 


Sch 3 


Sch 4 


Sch 5 


Sch 6 


Teacher report of frequency of collaboration on lesson 
plans and reviewing student work this year (mean) 


6-9 

times 


6-9 

times 


6-9 

times 


3-5 

times 


10 + 
times 


6-9 

times 


Source: Teacher survey 

Note: IVfean was calculated from an item with the following response options: never, once or twice, 3-5 times, 6-9 times, 10 or 
more times. 



In Schools 5 and 6, where we found strong evidence for teacher learning, teachers 
described meeting twice a week, once in an interdisciplinary team and once in their 
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departments. In these meetings, teachers worked collaboratively to write and refine 
curriculum, develop alignment across the curriculum, and create interdisciplinary 

projects. In addition to collaborative team meetings, the 
schools held bi-weekly whole- school professional 
development meetings and staff meetings. In follow-up 
interviews, 15 out of 16 teachers said that weekly 
collaboration within disciplines and in interdisciplinary 
teams was helpful to their practice. The network of 
Intermediary C schools in which these teachers worked 
also allowed educators to draw on the expertise and 
resources of educators across schools through formal 
and informal interactions, as described in Chapter 3. 

In Schools 1 and 2, where we found moderate to strong 
evidence of teacher learning, teachers described how 
weekly department meetings provided structures for 
collaboration. They spoke positively about the ways in which meeting times were used 
for collaboration, developing alignment across the curriculum, and professional 
development. In follow-up interviews, eight of eight teachers at School 1 and six out of 
eight teachers at School 2 spoke positively about their collaboration with other teachers 
in these weekly department meetings. A science teacher at School 2 reported, "We 
design lesson plans and share the types of projects we have done in class." An English 
teacher at School 1 described how the English department aligned the scope and 
sequence from 9 th through 12 th grades: "We meet as an English department to make sure 
it flows nicely from grade to grade. In 9 th grade we concentrate on thematic and 
character essays. In 10 th they concentrate on persuasive essays." A math teacher at 
School 1 described how weekly meetings provided opportunities for teacher 
collaboration such as discussions about howto incorporate accountable talk into the 
math curriculum. These examples, as well as those from Schools 5 and 6, indicate an 
emphasis on teacher collaboration and planning that is organized both vertically across 
grades and horizontally across disciplines, leading to a coherent and integrated 
instructional approach. 

In the two schools where we found less evidence of teacher learning, there were 
dedicated monthly meetings for teachers in each discipline, but some teachers reported 
that generally these meetings did not provide opportunities to develop and coordinate 
curriculum. In follow-up interviews, three out of eight teachers at School 3 described 
these department meetings as ineffective. For example, one teacher noted that, since 
department meetings were held only monthly, teachers had to build in collaboration 
time— often informally— outside these meetings. Four School 3 teachers mentioned in 
their interviews that they lacked time for collaboration. Teachers at both Schools 3 and 
4, when asked about opportunities for collaboration, frequently cited informal means of 
collaboration such as interactions in the teachers' room. Findings suggest that teachers 
sought the support of peers, but that, without regularly scheduled meetings where 
collaboration and learning were a focus, such occurrences were limited to pockets of 
teachers working together. 



"We coordinate the curriculum across grade 
levels. The 9-1 0 th grade science teachers have 
multiple meetings, and the 11 th grade science 
teacher joins the meetings, and we do planning, 
so the 9-1 0 th grade curriculum spirals into the 11 th 
grade. We also have interdisciplinary team 
meetings, and we host a science blog where we 
share a lot of planning tips and devise content 
themes for our classes. " — IVhth teacher, School 5 
(interview) 
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Culture of Reflection and Inquiry 

A part of effective teacher learning is related to how teachers use their time together. 
Effective professional communities are characterized by a "deprivitization" of practice 
(Little, 1990) as teachers come together to engage in 
reflective dialogue about teaching practices, to observe 
one another's classrooms, and to engage in joint problem 
solving (Bryk, Cambum, & Lewis, 1999). 

Teachers in the two schools with strong evidence of 
teacher learning described a culture of reflection and 
teacher inguiry in the professional community of the 
school. In an interview at School 5, one teacher reported: 

"As a team, we visit each other's classes and do 
observations. In our collaborative team meetings we 
discuss what we saw in each other's classrooms and make 
suggestions." We also observed a professional 
development committee meeting at School 5 in which 
teachers worked on a yearlong plan for school- wide 
professional development focused on developing teacher 
inguiry through such activities as using student portfolios to reflect on their practice and 
on curricular alignment. After the meeting, the principal explained that this work is 
linked to broader goals of engaging teachers to be reflective about their work with 
students and about their own professional development. 

This culture of reflection and inguiry was also evident in School 6 where, in a focus 
group, a teacher spoke to the ways in which teachers are continually working to improve 
practice: 

We spend hours looking closely at student work, trying to learn about their 
strengths. It is connected to our professional culture. We look across different 
grades and looked to see if interventions were making a difference. In our 
curriculum share meetings every week, we'll look at work and share. 

At the two schools with moderate to strong support for teacher learning, we saw in- 
school professional development sessions that helped to establish a culture of inguiry 
and provided opportunities for more experienced teachers to share best practices with 
newer staff. At School 2, where 81% of teachers had been teaching for 0- 2 years and the 
other 19% for 3- 5 years, the principal reported in her interview that she seeks support 
from Intermediary A for teacher learning opportunities to develop her young staff. The 
considerable effort to enhance teacher development at these schools drew on the 
expertise of coaches and other personnel from the intermediary. 

In Schools 3 and 4, inguiry teams, consisting of teachers and administrators focused on 
enhancing practice and improving the performance of struggling students, were 
supposed to be in place, but, when we attempted to observe inguiry team meetings at 
School 4, we found that they had been cancelled. An inguiry team member in an 
interview characterized the work of the team as "floundering." In a teacher focus group 
at School 3, a teacher described his participation in an inguiry team that had met the 



Teacher Learning through Study Groups 

In response to the varying skill levels of teachers 
and the changing needs of the growing school, the 
principal at School 6 restructured the weekly 
professional development meetings, designing a 
process that allowed teachers to self-organize 
around areas of professional interest. For 
example, a group of science teachers worked 
collaboratively with students to develop a 
curriculum for students with advanced science 
skills. In another group, drama teachers worked 
together to create a unit on integrating 
kinesthetic movement into all subject areas. 
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previous semester to monitor the progress of 16 students at the bottom third of the dass. 
The team's work involved observing students in classrooms, conducting student focus 
groups, and engaging in reflection about structures necessary to support students. Other 
teachers in the focus group responded to this teacher's description by indicating that 
they were not aware of the existence of inquiry groups in the school. Their reaction 
suggests that, though School 3 contained pockets of teacher learning and reflection, this 
work was not integrated into the overall work of the school. The structure of inquiry 
team meetings is designed to fadlitate collaboration and reflection, but teacher 
comments in these schools suggested that the teams had little institutional support. 



Teacher Participation in Design and 
Implementation 

In case- study schools with strong support for teacher learning and collaboration, 
teachers reported on in- school learning opportunities designed by teachers on 

professional development committees. Studies of effective teacher 
professional communities emphasize the importance of routines, 
such as regularly scheduled time for reflecting about teaching 
practices (Little, 2003). At Schools 5 and 6, a group of teachers 
served on a committee that worked to identify priorities and 
implement professional development. We observed a professional 
development committee meeting at School 5, comprising one 
representative from each of the instructional teams. The purpose of 
the committee was to plan professional development activities for the 
school and to communicate with the intermediary's professional development 
committee in order to influence the development of network professional development 
activities. The committee conducted a survey of teachers to determine their professional 
development needs. At School 5, to instill teacher- led professional development, the 
principal instituted a request- for- proposals (RFP) process in which teachers worked in 
cross- school groups to propose study groups on topics of interest. (See Chapter 3 for an 
example of such a study group.) 

The two schools with moderate to strong support for teacher learning showed less 
evidence of teacher participation in the design and implementation of professional 
development. In a follow-up interview, a School 2 teacher who had attended the 
intermediary's summer institute described a particularly useful "design your own 
workshop" session, which was intended to equip teachers to lead professional 
development activities. However, the data also suggest that the intermediary's coaches, 
rather than the teachers themselves, tended to lead teacher learning. In her interview, 
the principal of School 2 described the support that she received from the intermediary 
to redesign her advisory curriculum with the help of the intermediary's coach and of 
intermediary- sponsored workshops in which "teachers developed lessons plans 
according to the students' needs. Teachers 'own' these lessons; therefore they have a 
vested interest in implementing these advisory lessons." 



" I think that strength of our model 
is that it is shaped by teachers' input 
as opposed to the administration 
setting the agenda." —Teacher who 
represented School 5 on the 
intermediary's professional 
development committee (interview) 
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Distributed Leadership 



In addition to teacher collaboration, a culture of inquiry, and teacher participation, we 
also saw evidence of distributed leadership in schools with stronger support for teacher 
learning and more effective instructional 
practices. Scholars describe distributed 
leadership models in which teachers 
collectively shape and implement high-quality 
instructional practices and effective school 
structures (Copland, 2003). 



Shared Decision Making 



The two schools served by Intermediary C had 
a distributed leadership model that engaged 
teachers in shaping structures and practices in 
their schools. The intermediary used this same 
governance model to involve teachers in its 
own decision-making processes and to foster a 
pipeline of leaders that eventually move to new 
schools, bringing with them a deep 
understanding of the intermediary's model 
principles and practices. Teachers reported in 
focus groups that they appreciated the fact that 
the intermediary's professional development 
was collectively shaped and led by teachers. 
One math teacher at School 5 explained in his 
interview that he worked on the intermediary's 
professional development committee, which 
met with the organization's leadership 
monthly to design professional development 
activities and workshops. 



In May 2009, School 6 teachers, guidance counselors, and 
administrators participated in a two-day retreat to develop a 
guiding vision for the next five years. Staff members analyzed 
case studies of two current students to examine how current 
school structures addressed students' academic, linguistic, 
and social- emotional needs. Staff then were asked to envision 
the school receiving a maj or award in 2014 for their work in 
"educating the whole child.'' What structures and programs 
would need to be in place in order to win such an award? 
Based on their ideas, staff developed action plans and 
volunteered for groups that would work on various aspects of 
the plans in the summer and in the fall. Participants then 
brought their ideas back to the full staff in order to gather 
additional input and create buy-in. Examples of new projects 
included: 

Orientation. The first week of school would be devoted to 
intensive advisory activities focusing on study skills, learning 
group collaboration, and team building. 

College preparation in the 9 th and 10 th grades. Staff 
members agreed to develop plans for increasing the frequency 
and intensity of college prep activities, including one college 
visit a year, for 9 th and 10 th graders. 

Reintroducing advisory groups. Staff members decided to 
bring back advisory groups, which the school had not had for 
over a year. They agreed that advisories were critical to 
address students' social- emotional needs and to support 
students in setting school performance and college goals. 



In addition to the strong evidence we found for 
teacher learning at the schools served by this 
intermediary, we found evidence that the 
professional development model encouraged 

the development of leaders who can support the ongoing growth of the network. 
Teachers and administrators said in focus groups that conducting professional 
development workshops and working closely with intermediary personnel provided 
opportunities for leadership development. This model thus has a dual benefit: It not 
only ensures that professional development is aligned with teachers' needs but also 
increases the flow of institutional knowledge about successful practices across the 
network of schools. In the focus group of support staff at School 6, the assistant 
principal emphasized this leadership development: 

I have worked at three [Intermediary C schools]. . .. For each of these schools, I 
served as the representative from the schools on the [network-wide] 
professional development committee. It was the first leadership opportunity 
that I had. It gave me a chance to work with other teachers and to decide what 
kinds of professional development we would offer and facilitate. 
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In a focus group of eight School 5 teachers, one teacher suggested 
that leadership opportunities contributed to her overall feeling of 
connection and internal accountability to her professional 
community (see box) . In the professional communities of the two 
schools where we observed strong evidence for teacher 
collaboration and learning, we saw an emphasis on building 
capacity from within by engaging the collective knowledge of 
teachers through inquiry and shared decision making. 

The principal of School 1, an Intermediary A school, indicated in 
her interview that she saw distributed leadership as an important 
goal: 

My vision is about distributed leadership. I hope that [the current assistant 
principal] will take over when I leave. If she is not interested, then I hope 
someone else from our staff will be interested in taking that leadership role. We 
have four grade leaders and seven or eight department chairs. Two of our 
founding teachers actually left to start their own charter schools. 

Mention of opportunities for leadership was absent from interview responses of teachers 
in two schools. Schools 3 and 4, where we found limited evidence of support for teacher 
learning. Rather, teachers described barriers related to their own lack of authority. At 
School 3, teachers described feeling expected to "deliver" rather than having authority to 
shape and refine their own curriculum. A math teacher and a science teacher both 
described the curriculum as "set." The math teacher attributed this lack of freedom to 
Regents pressure: "Unfortunately, Regents and the state curriculum is "king."' Another 
teacher, on the other hand, attributed the "set" curriculum to the school administration, 
implying that he did not have the flexibility to align his curriculum with that of other 
courses in the same discipline: "The . . . curriculum has been set for the year— the 
grading, criteria, and exams are set. The administration wants us to have similar tests 
across the department." When another teacher was asked if he works to develop 
alignment across the curriculum, he responded, "I follow a comprehensive . . . syllabus 
that is closely aligned to the Regents." 

We saw similar evidence that School 4 teachers felt they did not have the authority to 
exert leadership. To explain why the inquiry team at this school was not meeting, a 
teacher said that, as a colleague/ peer teacher, he did not have "the authority to tell 
participants that they were required to attend these meetings." He added that the 
principal had not given him this authority. 

Taken together, teacher collaboration with a culture of reflection and inquiry, 
participation in the design and delivery of professional development, and distributed 
leadership seemed to foster a coherent instructional approach. The schools 
characterized by these elements also worked across grade levels and disciplines to plan 
and reflect on the curriculum and their instructional practices, ensuring coordination 
across grade levels so that classes built on previous classes toward college readiness. 



"All teachers can be part of 
subcommittees. It impacts your 
practices to be part of policy decisions, 
decisions about hiring, curriculum, all 
kinds of policies. It makes you grow 
because you're connected to your 
community. You're not an island who 
doesn't know what others are doing." 
—Teacher, School 5 (focus group) 
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Access to Resources 



Though it was outside the scope of our study, responses in teacher follow-up interviews 
and focus groups made it dear that the degree of access to resources could fadlitate or 
constrain teachers' abilities to design and implement high- 
quality curriculum in the case- study schools. 

In a chemistry dass we observed at School 6, a teacher had 
students create videos to gain a conceptual understanding of 
chemical mixtures (see Snapshot of Effective, Rigorous 
Instruction on page 65). In the follow-up interview, this 
teacher described how access to resources allowed her to 
design creative and engaging curriculum: "I am so glad we 
have the resources that we do. I was amazed when I started 
at [this school] and learned of all of the resources our school 
has. It really helps to make teaching a worthwhile, effective, and fun experience!" 

During a focus group at this school, teachers discussed the administration's support in 
providing the materials they needed to create curriculum. 

By contrast, in an observation at School 4, an English teacher had created an 
"experimental" course in which seniors read "the kinds of novels that these 
upperclassmen and women enjoy" and analyzed characters to whom students could 
relate. In the follow-up interview, this teacher explained that students had had to 
purchase their own textbooks for the course. She said, "One of the challenges for these 
lessons is that they don't all have the text. ... We are limited in what we can do and what I 
can assign." This teacher said that she copes with the lack of books by reading to the 
students. 

Our examples suggest that access to resources influences not only teachers' ability to 
create effective curriculum but also the degree to which teachers feel supported and 
valued in their professional communities. 



" No one asks me what I need for my 
classroom. This week they gave me an 
overhead projector. I don't need a 
projector.... I have no dictionaries. I put in an 
order in J anuary and I still don' t have any 
dictionaries. " —English teacher, School 4 
(focus group) 



Summary 

Data from multiple sources indicate that all of the case- study schools implement, to 
some extent, the effective instructional practices identified by the research. All of the 
case- study schools indicated that rigorous instruction was a priority and a prerequisite 
to preparing students for college. However, some schools seemed to implement effective 
practices, including academic rigor, to a greater extent than others. Not surprisingly, in 
classrooms where we observed frequent implementation of effective practices, we also 
observed students to be engaged in the instruction. The common element in the schools 
where we saw effective instruction and student engagement was a strong focus on 
teacher learning. 

Our data indicate that teacher learning is fostered by teacher collaboration and the 
provision of common planning time characterized by certain elements. These elements 
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indude dedicated time and space for formal collaboration that is guided by a 
professional community of practice— teachers working collaboratively to develop a 
sequenced and interdisdplinary program of instruction and engaging in continuous 
refledion on their practice in order to solve problems and improve instruction. The 
schools with stronger implementation of effective practices also emphasized distributed 
leadership and teacher involvement in their own learning. 

Taken together, these elements seemed to fadlitate a more integrated approach to 
instruction that was aligned both vertically, across grades, and horizontally, across 
disdplines. This alignment, in turn, enhanced teachers' focus on working toward college 
readiness. In these schools, intermediaries supported teacher learning by offering high- 
quality professional development opportunities in which teachers partidpated in both 
planning and implementation. The two schools with the strongest support for teacher 
learning were served by the intermediary that focused on fostering a strong 
collaborative network, both across and within schools, in which teachers and 
administrators could share expertise and resources. 



74 Small High Schools at Work: A Case Study of Six Gates-funded Schools in NYC 




6. College Preparation 



Getting students "college ready"— prepared to enroll and succeed, without remediation, 
in a credit- bearing postsecondary institution (Conley, 2007) — is a foremost goal of the 
Bill & Melinda Gates Foundation and of the intermediaries and schools in our study. 
Indeed, one intended outcome of the focus on personal and academic support (Chapter 
4) and on high-quality instruction (Chapter 5) is to prepare students for post- secondary 
education. 

Recent research has identified several key aspects of college readiness: Students need to 
have the cognitive strategies, content knowledge, self- management skills, and 
knowledge about postsecondary education to access and be successful in college 
(Conley, 2007). Cognitive strategies include those fostered by effective instruction, as 
discussed in the previous chapter. Additional strategies— habits of mind" (Conley, 
2007) essential to college-level work— are explored below along with self- management 
skills and knowledge about postsecondary education. The scope of this study did not 
include the extent to which schools developed content knowledge. However, the impact 
study conducted by MDRC will measure the impact of the Gates- funded small schools 
on student achievement in content areas as measured by the Regents exams. 



The case- study schools, to a greater or lesser degree, addressed college preparation by 
embedding college readiness and a college- going culture in virtually every aspect of 
students' educational experience. We saw the emphasis on college reflected in: 

• A school- wide college- going culture 

• College access activities 

• College readiness activities 

The distinction between college access and college readiness is a theme that 
emerged from teacher focus groups and interviews. College access refers to 
the activities necessary to get students into college— entrance exams, 
applications, financial aid, and the like. College readiness refers to the skills 
and abilities students need to succeed once they are enrolled. As a teacher 
noted in a School 3 focus group, "There's a difference between getting kids 
into college and getting them college ready. We get them into college, but 
then when they get to college they need to take extra classes and need extra 
help." 



"The primary focus of our 
school's leadership is to 
address the issue of getting 
students ready for college. " 
Teacher, School 1 (interview) 



School-wide College-going Culture 

Students served by the case- study schools, most of whom would be the first in their 
families to attend college, do not see necessarily themselves as being on a college track. 
For this reason, all of the six case- study schools hammered home, repeatedly and in 
multiple formats, the message that the school's role was to prepare its students for 
college. 

The college- going message began with the leadership and then was instilled in 
staff, students, and parents. Every one of the six principals said in the 
interview that preparing students for college is integral to the vision and 
mission of the school. For example: 

That's what I tell teachers. Remember our goal is to get students into college. 
(School 2) 

[Our mission is] to provide for the success of our students. Students of today 
must be able to adjust to the rapidly changing technology and society that we 
live in. This school will use all the resources available to prepare students for 
college. (School 4) 

In interviews, teachers confirmed the leaders' role in conveying the college message. 

The consistent message from the leadership has been for teachers to focus on 
college readiness. (School 2) 

My principal is always encouraging me to do professional development related 
to preparing students for college. (School 6) 

From the leaders and staff, the college- going message is in turn communicated to the 
students. Students at all six schools overwhelmingly reported in their focus groups that 



"A lot of students haven't 
even considered that they 
are on a college track. 
Repeating it a lot in class 
and advisory is very 
important." —Teacher, 
School 5 (focus group) 
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